This is a story about what happens when you give a Mechatronics engineer a new toy and a free Friday afternoon.

I’ve always been interested in sensors and how to use the information they collect. After we recently purchased some Arduino boards, I set about exploring different methods of manipulating the virtual world from the physical world. I was looking for an alternative input device to the keyboard and mouse.

After conquering the mandatory “Hello World” program on the Arduino, I started experimenting with some of the sensors. Simpler sensors include those that are sensitive to light, temperature, pressure, sound and movement. This project started with a thermistor (temperature sensitive) and an LDR (light sensitive).

Using the sensed data to manipulate a Rhino model. I used a piece of open source software called UbiMash produced by Flora Salim and her team at SIAL. UbiMash enables a connection between the Arduino board (physical world) and Rhino modelling package (Virtual world).

The following video shows the final product. The shape changes height as the light levels change, and the colour changes as the sensed temperature changes.

What could it be used for? Sculpting models in Rhino? Interactive public art display?

Arduino code (modified version of code supplied by UbiMash):

int ledPin=9;
int ldrPin=1;
int lightVal=0;
int tempPin=2;
int tempVal=0;
void setup()
{
Serial.begin(9600);
Serial.flush();
pinMode(ledPin,OUTPUT);
}
void loop()
{
// for (int i = 0; i < length; i++) {
lightVal=analogRead(ldrPin);
analogWrite(ledPin,lightVal);
delay(lightVal);
analogWrite(ledPin,0);
delay(lightVal);
//Serial.print(i);
Serial.print(“light:”);
Serial.println(lightVal);
//Serial.print(“1: “);
tempVal = analogRead(tempPin);
Serial.print(“temp:”);
Serial.println(tempVal);
}
Rhino Code (some code provided by UbiMash)

Option Explicit

Call Main()
Sub Main()

 
  ‘ Create a new square surface
  Dim sqSize : sqSize = 200
  Dim ptOrigin : ptOrigin = Array(-sqSize,-sqSize,0)
  Dim ptX : ptX = Array(sqSize,-sqSize,0)
  Dim ptY : ptY = Array(-sqSize,sqSize,0)
  Dim dX : dX = Rhino.Distance(ptOrigin, ptX)
  Dim dY : dY = Rhino.Distance(ptOrigin, ptY)
  Dim arrPlane : arrPlane = Rhino.PlaneFromPoints(ptOrigin, ptX, ptY)
  Dim plane : plane = Rhino.AddPlaneSurface(arrPlane, dX, dY)
 
  ‘ Add (and get) more control points to the plane
  Rhino.RebuildSurface plane, Array(2,2), Array(5,5)
  ‘Dim points : points = Rhino.SurfaceEditPoints(plane,False,True)
  Dim j
  Dim sLight, sTemp, resLight, resTemp
  For j=0 To 250 Step 1
    ‘Retrieve ARDUINO Light and Temperature Sensor data
    sLight = “ARD|LIGHT”
    sTemp = “ARD|temp”   
    resLight = getSensorData(sLIGHT)
    resTemp  = getSensorData(sTemp) 
 
    ‘ Print out all control points
    Dim points : points = Rhino.SurfaceEditPoints(plane,False,True)
    Dim i
    For i=0 To UBound(points)
      points(i)(2) = 500 – resLight
    Next  
 
    points(0)(2) = 0
    points(4)(2) = 0   
    points(12)(2) = (500 – resLight) \ 2
    points(20)(2) = 0   
    points(24)(2) = 0
 
    ‘ Move the surface points
    Rhino.DeleteObject(plane)   
    plane = Rhino.AddSrfPtGrid(Array(5,5), points)

    ‘ Change the colour of the object  
    Dim blueCalc, redCalc
    blueCalc = 255 – (255\30) * (resTemp-490)
    If (blueCalc > 255) Then
      bluecalc = 255
    End If
    If (blueCalc < 0) Then
      bluecalc = 0
    End If
   
    redCalc  = (255/30)*(resTemp-490)
    If (redCalc < 0) Then
      redCalc = 0
    End If
    If (redCalc > 255) Then
      redCalc = 255
    End If     
    Call Rhino.Print( “blue: ” & blueCalc & ” red: ” & redCalc)
    Rhino.ObjectColor plane, RGB(redCalc, 0, blueCalc) 

    sleep(125)

  Next 
 
  Rhino.DeleteObject(plane)   
 
 
End Sub

Function getSensorData(varName)

  Dim objPlugIn, objTest
 
  ‘Rhino.Print “Attempting to get ResponsiveRhino”

  ‘ Get the TestRhinoScript plugin’s
  ‘ primary COM visible object
  On Error Resume Next
  Set objPlugIn = Rhino.GetPlugInObject(“ResponsiveRhino”)
  If Err Then
    MsgBox Err.Description
    Exit Function
  Else
    ‘Rhino.Print “Successfully get ResponsiveRhino”   
 
    ‘Get the current light sensor value
    getSensorData = objPlugIn.GetTopicValue(varName)
         
  End If

End Function

Back in November 2009 we ran the first Arup HackDay. During HackDay people from around Arup gathered to create new tools and mash data over a 24 hour period. One of the tools created at HackDay by Andrew Tsakmakis was re-purposed to create the Secret Lives of Projects.

I’ve long been interested in project-based organisations and how new knowledge can be transferred from one project to another. The Secret Lives of Projects is a visualisation of the content created during an Arup project. It is a thought piece around the question “What is the shape of a project?” and an attempt to visualise some of the patterns of content creation as we work on projects.

The original tool was developed to index data with several features for quick searches. We did some more work after HackDay and pointed it towards a project directory. We then extracted meta-data from email archives and files. People in the email archive were identified and their business units and locations matched. We then linked billing data.

In the excerpt from communication network (below) the nodes reflect a pattern of communication through the project as people email each other. Nodes with yellow or blue dots grow as they bill time and so a picture emerges of who is speaking with whom and who are the key players working on the project. If this tool ran in real-time on your desktop could you tell if people were in the loop or not?

We worked with Greg More from OOM Creative to build an application using Adobe AIR and Flex. The video shows some of the visualisations from the AIR application.

Smartphone user waiting for tram, Sydney

This update is in three parts: ‘our approach’; ‘the hardware’; and ’sensing’. This entry is continued from ‘the hardware’.

It turns out few people have Bluetooth turned on and visible by default. When my colleague at Arup Jason McDermott helped create an urban sensing installation in central Sydney (Smart Light Fields) they observed around 8% of passers-by had Bluetooth on and visible. A colleague in Barcelona reckons it’s more like 20-30% around La Ramblas (is Bluetooth more prevalent in Europe due to a more mature applications market?).

Either way, we can only see a sample of passers-by with Bluetooth. So our approach is to then move through a stack of sensing, moving on to wi-fi (on smart-phones and other connected devices like netbooks), GSM and so on.

Wi-fi is particularly interesting, as we’d initially thought it wouldn’t register much – as the phone needs to be configured to auto-connect to networks, and a network and router i.e. a wireless access point (WAP), needs to be present to enable that. Even if we could prime the environment with a router (leaving aside the provision of amenities and apps approach – see later), we’d assumed a very small percentage of phones would be detectable. If you know the MAC address of a device, you can ‘ping’ it – but how to discover the MAC address? Leaving aside issues of privacy and ethics momentarily, this was proving a challenge just on a technical basis alone.

However, CRIN have been pursuing an approach called “packet injection” (which most wi-fi dongles don’t support, but some do). To cut a long story short, using this, you can prompt a wi-fi port to respond to and reveal its MAC address. It sounds somewhat scary – and there are those ethical issues to the fore about how we reveal the presence of this system, how to reveal its seams, and so on – but this could be extraordinarily powerful It would mean that all wi-fi-enabled devices i.e. most smartphones, all netbooks, could be triggered to reveal their MAC addresses. This initial process can take up to a minute – again, lending itself to environments where phones/devices are stationery or slow-moving, relative to movement at least – but once you have the MAC address, you can trigger it far more quickly.

A couple of sensors might be deployed here. One, in ‘wide range’ at an entrance, say, would be learning MAC addresses. The other might be tighter and more focused, pinging those addresses already discovered – this only takes a second or two once discovered.

GSM is more complex again, and we’ll report on that later (but in essence, unless you’re network provider you can’t trigger a phone to communicate (strictly speaking you can, but it’s illegal to), and so you rely on how often phone sends a ‘ping’, which it does when it’s crossing a cell location area boundary, when it’s on a call or SMSing, or every 2 hours 20 or so in Australia, depending on how the network configures such things.) With 3G the update periods are shorter, and there’s likely to be more data transfer occurring, which can also be detected. 4G, 3.5G (LTE/WiMAX etc.) are even more likely to be capable of being detected.

The next challenge then, other than refining both sensing processes, is to combine the Bluetooth and wi-fi sensing together to give an aggregate view of phones/devices across both. A lot of shuffling of pings and prompts and duplicates is required here. Beyond that, to get the hardware prototype to a state where we can test in a few environments – at UTS and perhaps at Arup, and then into the wild.

So now we have significant process in both the sensing side (Bluetooth and wi-fi both delivering results, albeit independently at the moment, and the hardware side, with a ‘box of tricks’ coming together nicely.

One of the key questions this might be able to answer are those deceptively simple questions about how people use transport. “How many people get off the number 12 bus here and change to the number 34?”, for instance. Although answers are usually attempted via spot-surveys and manual tracking, this very simple analysis is actually difficult to get at in any systematic sense – particularly when scaled up to all buses, all bus-stops, and in real-time. (The shift to real-time data driving decision-making in urban planning is fundamental, and not without complexities, but will surely happen. This project is a prototype of some elements involved in that future.) Other questions might be: How many people switch from platform 3 to 4 at this time? How many people are on-board the next bus? And so on.

There are issues around how to scale up the data to an urban population. As Anne Galloway and others remind us, not everyone has a mobile phone, never mind a smart-phone. So although we’re investigating a mode of sensing mobility that would be far more wide-spread and real-time than current methods (leaving aside floating car data) we have to be careful about extrapolating the results. We also have to be careful about ethics and privacy too, and we’ll be investigating that shortly, covering both the visibility of the seams of the system, and the levels of aggregation necessary to preserve anonymity in visualisation. As Richard Sennett has said, the great promise of cities is that they enable both anonymity and community – we want to work with that balancing act, not against it.

All the above assumes a ‘passive’ sensing of what phones/devices are present in a space i.e. no direct intervention into the environment encouraging access. However, if there was a smart transit application built for these environments – say delivering real-time transit information, including arrival/departure times, connection times and possibilities, congestion levels, environmental data etc. – this would give people a reason to turn on Bluetooth and/or wi-fi on in the first place. If transit operators offer a good quality informational amenity to passengers, the passengers’ quality of service is improved, potentially leading to great patronage – but in addition, these users can then also be sensed, offering the kind of strategic data we’ve been discussing to transit operators.

As a side-line, we’re also looking at office environments, and how responsive environments could be enabled by these real-time feeds on presence. Many ideas there, which we’ll pick up in a subsequent post.

David Lowe and Jason McDermott

This update is in three parts: ‘our approach’; ‘the hardware’; and ’sensing’. This entry is continued from ‘our approach’.

CRIN have explored some simple hardware platforms, particularly the Gumstix and Beagleboard ’sawn-off’ PCs (strictly, ‘computers-on-module’). Below, a few shots from a recent session discussing the hardware. Be warned, this will get a little geeky.

Note the core board here is a Gumstix Overo Earth – the smaller component – sitting on top of a larger expansion board, which gives us further connectivity, including an outlet for multiple USB connections amongst others (spec here). This is needed as CRIN are taking the ‘array of dongles’ approach to sensing via four Bluetooth dongles all scanning simultaneously. (You can see a couple of those behind the box in the shot below.)

Note also a 3G dongle is needed to deliver data from the board to the internet (and to enable updates to the software to be received on the board too).

There is also a wi-fi dongle, ready for the next stage of scanning (notes on that to follow).

The power adaptor, or battery pack, as seen here, is by far the biggest component. If the boxes are installed where reliable, secure power is available, then the box can be rationalised heavily, down to not much larger than the augmented Gumstix itself. In many cases – most urban bus-stops, public transport vehicles themselves, offices etc. – this power will be available. In far-flung suburban bus-stops, it may not be. So the battery approach isn’t ideal, but may be necessary in some environments. (Also, CRIN (David Lowe, Alex Gibson) reckon the program code is running at about 7% CPU at the moment. By rationalising other aspects of the software, the power load can come down even more. However, Alex is exploring running a background task to keep the 3G connection open (those service plans are not particularly designed for this, of course.)

CRIN can also wirelessly update the software on the boards, when required, by SSH-ing onto the card via their IP address (using Dynamic DNS, which avoids the issue of IP addresses changing regularly due to DHCP, usually used when connecting through an ISP like Dodo. Dodo have been chosen as the 3G connectivity as they’re one of the few to offer a long term pre-paid plans. Again, it’s interesting how these ancillary services can inadvertently shape the research. Though bandwidth not an issue here, due to the tiny packets of data being transferred.)

The data is being updated to Pachube. The problem here is that Pachube only currently supports 15 minute updates, and we also require a broader longitudinal history than it enables. Both of these aspects are being addressed by Pachube though, so we see great value in continuing to patronise that platform with our data (and here it is, for what it’s worth at this pre-installation point). The memory card on the board – a basic micro-SD card – has more than enough room for years worth of data stored locally, but getting the data into ‘the cloud’ via Pachube makes it far more malleable.

CRIN are going to test how scalable this approach is – say, by putting seven Bluetooth dongles on a Gumstix board, which are then scanning at slightly different start times. Results across all seven could be aggregated, and with duplicates removed, give a significantly better return in terms of phone detection.

The total cost of the box (all internals, connectivity etc.) is looking like it’s around $400 at this point. Note, at this stage, the boxes are being chosen for their robustness and unobtrusiveness. At a later date we will consider the visual design and affordances of such boxes, as the need to convey the existance of this system might figure for ethical and adaptive design reasons. Besides, why not make a nice box?

We’ll consider next steps, and the other sensing approaches in the ’stack’ in the next post.

Mobile phone user on CityRail, Sydney

This update is in three parts: ‘our approach’; ‘the hardware’; and ’sensing’.

A quick technical update on our mobile phone sensing project with UTS (see earlier post for context). This project is exploring technical approaches to sensing the presence of mobile phones in transit environments (bus, train, ferry etc.) as well as pedestrians, in order to provide real-time data on such activity, potentially informing urban planning and transport planning decisions. Such approaches might reveal how the city is being used, in real-time. This write-up will get a little geeky in places, but we share it in the hope you’ll find something interesting in the overall idea or the particular approach, and do feel free to contribute via the comments form at the bottom of each post. We’re interested in your feedback.

Our colleagues at the Centre for Real-Time Information Networks (CRIN) at UTS have made significant progress in terms of both the sensing process and the hardware prototypes.

Dealing with the first part, we’ve been exploring a ’stack’ approach to sensing phones, starting with scanning for Bluetooth, then wi-fi, then GSM, and so on. This is partly due to ease of sensing, and partly exploring ethical issues i.e. if people have Bluetooth turned on, or wi-fi connecting to routers automatically, can we assume they are more likely to be happy to be sensed? (Probably not, due to poor design on the part of mobile phone software meaning many may leave it on by default without paying much attention to it thereafter, but part of the point of the research is to explore these issues of privacy and security as well as technical approaches.)

And dealing with the first of those wireless technologies, CRIN have made particular progress in terms of sensing Bluetooth. Using the basic Bluetooth scanning functionality in a PC or Mac Mini, say, we can sense people with Bluetooth turned on and visible if they’re walking past slowly, due to the relatively slow default scan rate i.e. it takes a while for the scanners to detect and observe the phones in the vicinity (the scan rate takes over a second, and is dependent on the number of devices. In essence the scanning uses multiples of 1.28 seconds, with the number of multiples increasing the liklihood of finding all devices. A good quick summary can be found in this PDF.)

As we’re trying to spot a couple of things – for example, both passengers in transit or waiting at a bus-stop (more static) and also pedestrians (moving at around 1-5m/s) – and given the likelihood of groups in these scenarios and the low numbers of people scannable, we needed to increase the Bluetooth scan time.

There are legal and illegal ways to do this. Choosing the former route, CRIN have made great progress in terms of speeding up the scan time, and the detection rate. Software is being written in Python, on the Linux operating system – rather than say Processing on Mac OSX, where the need to parse higher-level languages with limited direct interfaces between Bluetooth drivers and Java would slow things down a little – and several hardware approaches have also been explored, with the current solution considering using multiple Bluetooth dongles in an array, staggering scan times

The range of Bluetooth (class one) effectively turns out to be around 5-20m (depending on the particular dongles employed, the structures in that environment, and so on. Wi-fi is much broader). Of course, in transit, on a bus, train, or tram, or relatively stationery at a bus-stop/platform, the captive audience is much easier to spot.

(NB: the RTA assumes people walk about 1.2 m/s on average, according to their transport planning regulations.)

Essentially, the array of Bluetooth dongles is now able to scan phones much faster, and certainly within our intended environments of buses, trains, bus-stops, platforms, stations etc. Recall that the original rationale for this project is to generate real-time feeds on transit activity in urban areas, as most current transit data is not real-time, not particularly scalable, doesn’t uncover individual ‘multi-modal’ trips where someone might walk to a bus-stop and then switch to a train, say.

Given this impetus, the scan-rate from the Bluetooth array achieved above is certainly good-enough as a start. The next requirement is to wirelessly communicate this data in real-time to ‘the cloud’, via a small robust ‘box’ that could be installed in such environments.

In the next post, we’ll discuss the emerging hardware prototype.

The DesignLink Software Development toolKit (SDK) is now open via a collaboration agreement. DesignLink is designed to assist in the development of software to solve interoperability problems between various programs used in the built environment (AEC) design process.

Here is a link to an earlier post and DesignLink’s ArupForge page lists some of the software packages and processes we’ve already tackled. Its open now to extend and contribute.

Following are two videos of DesignLink in action. The first shows the use of Octave/Matlab to drive the generation of EnergyPlus and Radiance files from a GenerativeComponents model. Then Paul Jeffries from Arup AGU shows his Rhino3D to GSA tool called Salamander.

Contact us for further information.

We have started tweeting. The link above is a word cloud generated from the bios of our followers. Follow us @ arupaustralasia

Late last year, a colleague in Melbourne asked if I knew of a way to animate a Sankey diagram. A Sankey what? thought I, not within reach of Wikipedia (here’s the link). We then had a discussion on the flows of energy from generation to consumption and how such a diagram would transform its shape according to the situation.

In a report, and as a static diagram, the Sankey works quite well. But if its part of a story of varying input and modifying consumption then the diagram has a narrative along which it can contort itself. As we flicked through various technical reports I was thinking this diagram needed to jump off the page, tell the story and connect the various measures by which it is being varied to show the full picture.

So, if we’re working on an urban masterplan and reducing the consumption of electricity, what do those measures look like and how do they impact upon the total system. It didn’t really take all that long to piece together the various elements to make a story board for an animation. Three reports in fact; then some internal investment and the deft graphical hand of Fooch from Squint/Opera (Aus).

In the film, (excerpt below – we’re polishing the narrative) the content and the story are from our reports. We know though that not everyone reads technical reports, so this is the beginning of what’s next – visual reports.

For the past two years I’ve been part of an Australian Government funded research team exploring the theme of Delivering Digital Architecture in Australia, investigating issues such as architect/engineer communication, data interoperability and lessons from parallel industries.

Early in the research we gathered information about data interoperability experiences from our staff and collated it into an Interoperability Database. This revealed some basic approaches to interoperability problems:

Use an import/export tool written by a software vendor.
Write your own tool to do the data transformation Depending on the particular tool, these approaches can be characterised as either a Scalpel or a Swiss army knife.

Scalpel A tool that links two particular pieces of software (e.g. GSA to Revit)

Pros:
• Specifically designed for the task, often giving the best results.
• Can be optimised to deal with cross-discipline or “geometry + analysis data” transfer.

Cons:
• Can limit the choice of design tools.
• Causes problems as the number of design tools increases.
• Some user written tools prove difficult to share with others.

Swiss Army knife A more general purpose approach, using an exchange file format such as IFC.

Pros:
• Generally industry funded, based on well designed and documented formats.
• Often provided by software vendors as a feature in their products.

Cons:
• Might not meet project data requirements, and may not be easily extendable.
• Currently best suited to “documentation/co-ordination” data rather than “design/analysis” data. As part of our research we are proposing a third possibility, which solves some of these problems but, as with any solution, also has its downsides.

The DesignLink Software Development Kit (SDK) – a meta tool for building tools.

The DesignLink SDK combines a data exchange format with the routines needed to read/write that data to various applications or file formats.

The DesignLink SDK can be used to build tools to solve interoperability problems as well as custom tools to use in the design process. It’s this ability to use the SDK as a platform for building custom design tools that sets it apart from other data exchange toolkits. Since these custom tools share some common DNA, it will be easier for programmers to share useful code. They also benefit from being built on a platform that includes:

• inbuilt testing routines
• easy methods for people to contribute ideas or code, without needing a degree in software development
• a community of programmers and users to review and improve the code

The types of tools that can be built with the SDK include programs to enable designers to connect parametric geometry models to engineering analysis models, allowing them to explore more design options; and custom written programs that perform design processes such as member size optimisation and design code checking.

We recently showcased DesignLink at SmartGeometry 2009 and I’m currently in the process of testing DesignLink on project and planning the next steps, which might involve Collaborative Licensing to further develop the SDK.

We have been developing MassMotion, a unique 3-D simulation software application that can be used to predict human behaviour within virtual built environments. The people, or agents, in MassMotion models are intelligent and can retain information about their virtual environment. The agents are able to recognise congestion and to consider alternate routes based on their familiarity with an environment. The inclusion of the kinds of emergent phenomena that occur every day in the real world – such as lighting – allows us to analyse crowd behaviour in all kinds of buildings, transit areas and public spaces and landscapes.

The really nice feature the 3-D environment in MassMotion offers is the ability to import design geometry so with very-early-stage schematic models, designers can watch and observe how a space might perform and then make changes to the proposed space before running it again.

MassMotion was originally developed for transport planning by a team in Arup’s Toronto office, and has now been adopted by Arup’s pedestrian planning team in Australia. During 2008, Arup funded a project called Extending MassMotion to investigate how other engineering, planning and design disciplines could add information to models about phenomena that influence human movement. During this period, we spoke with clients and academics to further develop the research and business applications of MassMotion, and developed ways to better communicate the work – for example, using software engines and peripheral devices such as Nintendo Wii controllers to navigate the environment.

One of the outcomes was to introduce influence maps, as above. These can be used where there is not hard items defining a path like furniture or walls, but transient elements such as smoke.

If you’d like something to print click the image above for a pdf on MassMotion