Add a live feed

Rather than building a live-data ingest application from scratch, you will import a Streams project that has already been prepared. This application uses an operator called HTTPGetXMLContent to connect to a web services feed from NextBus.com and periodically (every 30 seconds) download the current locations, speeds, and headings of San Francisco Muni’s buses and trams. That operator comes from a version of the com.ibm.streamsx.inet toolkit that is available only on GitHub. This toolkit was installed in your environment when you installed the lab files.

The application parses, filters, and transforms the NextBus.com data and makes the result look similar to the file data although some differences remain. It exports the resulting stream with a set of properties that match the subscription of your processing application. When you launch the NextBus application, the connection is automatically made and data flows continuously until you cancel the job.

To add a live feed:

  1. Before you can use the NextBus project, tell Studio where to find the version of the com.ibm.streamsx.inet toolkit that it depends on:
    1. In the Streams Explorer, expand IBM Streams Installations [4.2.0.0] > IBM Streams 4.2.0.0 > Toolkit Locations.
    2. Right-click Toolkit Locations and select Add Toolkit Location.
    3. In the Add toolkit location dialog, click Directory and browse to My Home > Toolkits. (My Home is at the top; the dialog starts in the separate Root tree.) Select Toolkits and click OK.
    4. Click OK
      If you expand the new location (Local) /home/streamsadmin/Toolkits, you see com.ibm.streamsx.inet[2.7.4]. This is different from the version of this toolkit that is installed with Streams (2.0.2), so the NextBus project can select the right one by version. The 2.0.2 version is under the location STREAMS_SPLPATH.
  2. Import the NextBus project:
    1. In the top Eclipse menu, click File > Import.
    2. In the Import dialog, click IBM Streams Studio > SPL Project. Then, click Next.
    3. Click Browse and in the file browser, expand My Home and select Toolkits. Click OK.
    4. Select NextBus and click Finish.
  3. Expand project NextBus and namespace ibm.streamslab.transportation.nextbus. Launch the application NextBusIngest. You might need to wait until the project build finishes.
  4. Maximize and organize the Instance Graph. If you want, you can expand the nested composites in the NextBusIngest job.
  5. Check your results.

You should see the three applications that are connected. Tuples flow from the FileIngest job as long as you keep the “Infinite source” process running. Tuples flow from the NextBus job in 30-second bursts. The error.observations file gradually fills with records from NextBus. Their vehicle IDs do not conform to the “Cnnn” format.

Refresh the error.observations file periodically. To refresh the file, click in the editor showing the file and click Yes in the File Changed dialog, which appears when Studio detects that the underlying contents have changed.