Print bookPrint book

Guide for Experimenters

This guide help you in questions related to the Experiment management tool and IoT-Registry API for advanced experimenters. 

Site: FIESTA-IoT Training Platform
Course: Guide for 3rd Parties
Book: Guide for Experimenters
Printed by: Guest user
Date: Monday, 11 December 2017, 8:53 AM

1 Experiment Management

This guide explores three important areas to help the new Experimenters in FIESTA-IoT. You can find relevant information about:

  1. Experiment Management
  2. Experiment Registry Management (ERM) API, and
  3. IoT-Registry API.

1.1 Experiment DSL (FEDSpec)

FIESTA-IoT experiment management is facilitated by the usage of a descriptive language (DSL) that is capable of hosting all the experiments of a specific User. This language is called FIESTA-IoT Experiment Description Specification (FEDSpec) and is depicted in 1st Figure.

FEDSpec, as shown in the figure, can host all the defined experiments of an Experimenter by including multiple FIESTA-IoT Experiment Model Objects (FEMO). FEMO is responsible of holding the description of a single experiment and consists of:

  • The description: (Optional) where a short textual description of the experiment can be added. 
  • The Experiment Design Metadata (EDM): (Optional) where the graphical metadata of the experiment editor can be stored (i.e. node-red) that facilitate the conversion of a FEMO to a graphical representation of the experiment to an Editor (these metadata are editor specific).
  • The domain of interest: where we can have the list of domains of interest of the experiment, which can be used for discovery purposes, based on the M3-lite taxonomy.
  • The FIESTA-IoT Service Model Object (FISMO): which is the main and most important experiment descriptive entity, one or many of which can be included in a FEMO object and thus create a complete experiment. 

FISMO, as shown in 2nd Figure above, consists of the following entities:

  • The description: (Optional) where a short textual description of the experiment's Service can be added.
  • The discoverable: (Optional) here a Boolean defines if the experiment is discoverable or not.
  • The service: (Optional) here the URL for accessing the “iot-lite:Service” can be hosted for providing access to the semantic datasets.
  • The experiment control: which controls how the Service should be processed; in particular, it specifies the conditions under which the service should be invoked (e.g., specifying a periodic schedule, defining specific visualizations, etc.). This control is facilitated by the following entities (shown in 3rd Figure below):
    • Scheduling: (Optional) which may be defined to specify a periodic schedule for query execution for a specific subscription. The schedule of the Service could be defined, if required, by identifying:
      • Start time: (Optional) the time that the service should be started.
      • Periodicity: (Optional) how often the service will be executed.
      • Stop time: (Optional) when the service should be stopped.
    • Trigger: (Optional) the URL that should be invoked in order to trigger the execution of the Service if required
    • Report if empty: (Optional) If true, a Result Set is always sent to the subscriber when the query is executed. If false, a Result Set instance is sent to the subscriber only when the results are non-empty.

  • The experiment output: (shown in 4th Figure below) which defines the required information for the instantiation of the output of an experiment. The output could be provided in various ways i.e. visualization (widget) at a webpage or as a file. The output configuration is facilitated by the following entities:
    • The location: which hosts the URI where the output should be sent.
    • The file: (Optional) where the file type of the output can be identified. The current valid list of files types that are supported are

-      "text/plain",

-      "text/tab-separated-values",

-      "text/csv",

-      "application/sparql-results+json",

-      "application/sparql-results+xml",

-      "application/sparql-results+thrift",

-      "application/json",

-      "text/xml" and

-      "application/xml"

    • The widget: (Optional) which defines the required information for the instantiation of the presentation GUI as defined at the initial user request at the experiment definition time. For example, the widget that is going to be used for representing the data like a speedometer, a spreadsheet, a map or a diagram.

  • The query control: is responsible to host the description of the required query that should be executed in order to provide the results/data from this experiment Service. Along with the query itself it holds additional entities that enable the discovery and dynamic configuration of the query. The query control configuration is facilitated by the following entities (shown in 5th Figure below):
    • Quantity kind: (optional) a list of quantity kind URIs, based on the m3-lite taxonomy, to identify the type of measurements involved in this query.
    • Static location: (optional) the geo-coordinates of the experiment/query that will facilitate the experiment discovery.
    • Query interval: (optional) provides the time interval that the query should collect data from by explicitly specifying the start/stop date/time (fromDateTime - toDateTime). Alternatively it can hold the duration in seconds from the present to the past in order to dynamically calculate the start/stop date/time. The latter is used if the experiment wants to get at each execution the values of the last X hours, X days etc.
    • Query request: this entity is the most important entity of the FISMO object and holds the W3C query data (SPARQL) that should be executed in order to retrieve the results of the experiment Service.
    • Dynamic attributes: (optional) this entity provides the ability to dynamically update attributes of the query at the runtime of an experiment (i.e. in a mobile application to provide the CO2 level of the current location the experiment geo-coordinates should be updated in each experiment execution). Dynamic attributes provides predefined attributes (the predefined dynamic attributes) and experimenter defined attributes (list of dynamic attribute) shown below:
      • Predefined dynamic attributes: (optional) it provides a list of dynamic attributes with a predefined name and a user defined initial value. These attributes currently include the:
              • dynamic query interval which further contains:
                • fromDateTime,
                • toDateTime and
                • intervalNowToPast which is defined as seconds from now to past. If this is set then fromDateTime and toDateTime are not used.
              • dynamic geo-location: This contains latitude and longitude information in the string format.
      • Dynamic attribute: (optional) it provides a list of dynamic attributes with a user defined name and a user defined initial value.
  • The rule: (optional) which hosts a rule that should be applied on top of the service results. The rule configuration is facilitated with the help of the following entities (shown in 6th Figure below):
    • The name: a textual representation of the applied rule name
    • The rule definition: a textual representation of the applied rule code which can be a Jena rule or a SPARQL CONSTRUCT.
    • Domain Knowledge: the URI stating the domain knowledge of the rule
    • Quantity kind: a list of quantity kind URIs based on the m3-lite taxonomy.

1.2 Define an experiment through Domain Specification Language (DSL)

Currently EEE or the Experiment Execution Engine does not support all the attributes available within DSL. Within FISMO, we only support name, description, discoverable, scheduling, reportIfEmpty, filetype, experiment output, dynamic attributes, widget and query attribute. Currently, we do no support service, trigger, and rules. Poll Now option in the Experiment Management Console is a type of on- demand trigger that we provide to the experimenters.

Experimenters should consider replacing “#*#” with experiment requirements. For simplicity we provide a template of FISMO that is supported by the EEE. 

<fed:FISMO name="#NAME#">

 <fed:description>#DESCRIPTION#</fed:description>

 <fed:discoverable>#DISCOVERABLE#</fed:discoverable>

 <fed:experimentControl>

  <fed:scheduling>

   <fed:startTime>#STARTTIME#</fed:startTime>

   <fed:Periodicity>#PERIODICITY#</fed:Periodicity>

   <fed:stopTime>#STOPTIME#</fed:stopTime>

  </fed:scheduling>

  <fed:reportIfEmpty>#REPORTIFEMPTY#</fed:reportIfEmpty>

 </fed:experimentControl>

 <fed:experimentOutput location="#URLLOCATION#">

  <fed:file>

   <fed:type>#FILETYPE#</fed:type>

  </fed:file>

  <fed:widget widgetID="eu.fiestaiot.analytics.toolkit">

   <fed:presentationAttr name="requestBody" value="#WVALUE#"/>

  </fed:widget>

 </fed:experimentOutput>

 <fed:queryControl>

  <prt:query-request>

   <query><![CDATA[

#[1/1] visualization type: 'Gauge' and sensors

#QUERY#

]]>

   </query>

  </prt:query-request>

  <fed:dynamicAttrs>

   <fed:predefinedDynamicAttr>

    <fed:dynamicQueryInterval>

     <fed:fromDateTime>#FROMDATETIME#</fed:fromDateTime>

     <fed:toDateTime>#TODATETIME#</fed:toDateTime>

     <fed:intervalNowToPast>#INTERVALNOWTOPAST#</fed:intervalNowToPast>

    </fed:dynamicQueryInterval>

    <fed:dynamicGeoLocation>

     <fed:latitude>#LATITUDE#</fed:latitude>

     <fed:longitude>#LONGITUDE#</fed:longitude>

    </fed:dynamicGeoLocation>

   </fed:predefinedDynamicAttr>

   <fed:dynamicAttr name="#DA_NAME1#" value="#DA_VALUE1#"/>

   <fed:dynamicAttr name="#DA_NAME2#" value="#DA_VALUE2#"/>

  </fed:dynamicAttrs>

 </fed:queryControl>

</fed:FISMO>



To further explain,

  • #NAME#: should be the name of the FISMO that experimenter want to have. For example:

<fed:FISMO name="2ndUseCase">

  • #DESCRIPTION#: should be the description of the FISMO to understand what is the FISMO is about. For example:

<fed:description>Over time all noise observations for a given location</fed:description>

  • #DISCOVERABLE#: should be either “true” or “false”. As explained before this attribute is used by the EEE to know whether the FISMO can be shown in the “Other Available FISMO IDs for subscriptions” tab in the Experiment Management Console. Using this other experimenters could reuse the query and scheduling information (subscribers should giving a new location that would send the results to them). For example:

<fed:discoverable>true</fed:discoverable>

  • #STARTTIME#: it is the time when the scheduling should start. In case, the #STARTTIME# in the past, the current time will be used and in case #STARTTIME# is in future, the given time will be used by the EEE to schedule the FISMO. The #STARTTIME# should be in DATETIME format.

<fed:startTime>2016-11-08T18:50:00.0Z</fed:startTime>

  • #PERIODICITY#: is the period after which the EEE should re-trigger the execution of the FISMO. It is in INTEGER format and denotes the Seconds. For example:

<fed:Periodicity>250</fed:Periodicity>

  • #STOPTIME#: it is the time when the EEE should stop executing the FISMO. In case, the #STARTTIME# in the past, and is less than #STARTTIME# an error is raised by the EEE. Thus it is advisable that the #STOPTIME# is in future and is greater than #STARTTIME# The #STOPTIME# should be in DATETIME format. For example:

<fed:stopTime>2017-11-08T18:49:59.0Z</fed:stopTime>

  • #REPORTIFEMPTY#: it is the Boolean value (“true” or “false”) that should be set in order for the experimenter to not to receive empty resultsets obtained after the execution of the query. A value false mean that if the resultset was null then do also report it. By default the value is true. For example:

<fed:reportIfEmpty>false</fed:reportIfEmpty>

 It should be noted that for some type of queries there always exist a result. For example, queries related to count are bound to return results even though it is 0. In such cases, reportIfEmpty essentially does not make any sense and results will eventually be sent back to experimenters.

  • #URLLOCATION#: it should be the location where the results of the query should be returned. This is usually a valid URL location. Please note that an experimenter cannot update this parameter once the FEDSpec is submitted. It is assumed that the experimenter has developed the functionality behind this link where EEE can upload the results in a “multipart” files format. In the current implementation of EEE, it is also assumed that this is a REST based API that implements HTTP POST with following REQUEST parameters:

connection : keep-alive
Content-Type: multipart/form-data; boundary=--timestamp

An example URLLOCATION is as below:

<fed:experimentOutput

location="http://ExperimentServer.org/store/"></fed:experimentOutput>

Currently, if the experimenter is using HTTPS, then they should use Letsencrypt certificate for this API/URL. All other certificates will fail. However, HTTP will pass through. This is because of the JVM not having all the certificates installed.  

  • #FILETYPE#: this parameter defines the response content-type. As described above valid content-types that are received are "text/plain", "text/tab-separated-values", "text/csv", "application/sparql-results+json", "application/sparql-results+xml", "application/sparql-results+thrift", "application/json", "text/xml" and "application/xml". The Experimenter should choose either one of them. Please note that we do not set the extension of the file itself. For example: 

<fed:file><fed:type>application/xml</fed:type></fed:file>

In case you are using widget for calling FIESTA–IoT Analytics toolkit, the result will always be in CSV format. Thus, this value will be overridden and results obtained from ERS will always be in CSV format.

  • #WVALUE#: is a JSON string that is of form.

{
  &quot;Method&quot;: [&quot;Method_1&quot;,_&quot;Method_2&quot;,&quot;Method_3&quot;],
  &quot;Parameters&quot;: [&quot;Parameters_1&quot;, &quot;Parameters_2&quot;, &quot;Parameters_3&quot;]
}

Note the &quot; The &quot; here is used to correcto parte the ". Setting this value will enable EEE to know if the FIESTA-IoT Analytics has to be executed or not. If the experimenters do not want this then they simply just not provide the widget tag. Further, if this is set then the query should be of specific format as well. Please refer to the query template below:

Prefix …
Select distinct ?sensingDevice ?dataValue ?dateTime
Where {
    …

}

Please note the SELECT statement. As per the guidelines set by FISTA-IoT Analytics tool, this SELECT statement should not be modified. The experimenters will receive a CSV file always if this is set. Setting the #FILETYPE# will have no effect. Also note that the permissible values of “Methods” and “Parameters” are:

 

 

An example “#WVALUE#” is as shown below:

{
 &quot;Method&quot;: [&quot;fft&quot;, &quot;linReg&quot;],
 &quot;Parameters&quot;: [&quot; &quot;, &quot;Predict&quot;]
}

Note that the values in Parameters should have one-one mapping to values in Methods. Further the “ is represented as &quot; to facilitate xml parsing.

  • #QUERY#: is the actual SPARQL query that should be executed by the EEE on the Meta-Cloud.  For best results we advice not to provide following query and be specific to the needs.

 select * where {?s ?p ?o.}

The above query would hinder the performance of the EEE. A number of valid queries that we have used so far in all our internal components and in-house experiments can be found in section 4.4. Note that the #QUERY# should be between “<![CDATA[#[1/1] visualization type: 'Gauge' and sensors” and “]]”.

The following parameters are to be used when dynamic attributes are needed in the query. 

  • #FROMDATETIME#:  This attribute represents the “from” date time a query has to be executed. Within the query please use “%%fromDateTime%%” so that it is replaced with the value provided correctly. This attribute is dynamic so this attribute should be presented as a default value in DATETIME format in the FEDSpec. In the FEDSpec, experimenters can define it as, for example:

<fed:fromDateTime>2006-05-04T18:13:51.0Z</fed:fromDateTime>

  • #TODATETIME#: This attribute represents the “to” date time a query has to be executed. Within the query please use “%%toDateTime%%” so that it is replaced with the value provided correctly. This attribute is dynamic so this attribute should be presented as a default value in DATETIME format in the FEDSpec. In the FEDSpec, experimenters can define it as, for example:

<fed:fromDateTime>2006-05-04T18:13:51.0Z</fed:fromDateTime>

  • #INTERVALNOWTOPAST#: This attribute represents seconds from now to past. This is used in case experimenters do not want to specify #FROMDATETIME# and #TODATETIME#. This attribute is not needed to be represent in the query, however in the query, experimenters should still have “%%fromDateTime%%” and “%%toDateTime%%” so that EEE can process the #INTERVALNOWTOPAST# and replace “%%fromDateTime%%” and “%%toDateTime%%” accordingly. EEE resolves “%%fromDateTime%%” as “current time - #INTERVALNOWTOPAST#”. In the FEDSpec, experimenters can define it as, for example:

<fed:intervalNowToPast>300</fed:intervalNowToPast>

  • #LATITUDE#: This attribute represents the “latitude” in a query that has to be executed. Within the query please use “%%geoLatitude%%” so that it is replaced with the value provided correctly. This attribute is dynamic so this attribute should be presented as a default value in float format in the FEDSpec. In the FEDSpec, experimenters can define it as, for example:

<fed:latitude>46.52119378179781</fed:latitude>

  • #LONGITUDE#: This attribute represents the “longitude” in a query that has to be executed. Within the query please use “%%geoLongitude%%” so that it is replaced with the value provided correctly. This attribute is dynamic so this attribute should be presented as a default value in float format in the FEDSpec. In the FEDSpec, experimenters can define it as, for example:

<fed:longitude>46.52119378179781</fed:longitude>

  • #DA_NAME_NUMBER# and #DA_VALUE_NUMBER# these attributes go hand in hand. They form a key value pair. It is advised that experimenters use the #DA_NAME_NUMBER# in the query as “%%DA_NAME_NUMBER%%”. Please note change of “#” to “%%”. In the FEDSpec, experimenters can define it as, for example:

<fed:dynamicAttr name="qk" value="http://purl.org/iot/vocab/m3-lite#AirTemperature"/>

<fed:dynamicAttr name="unit" value="http://purl.org/iot/vocab/m3-lite#Degree"/>

Note that in the example #DA_NAME1# is “qk” and DA_VALUE1=http://purl.org/iot/vocab/m3-lite#AirTemperature

A sample query that utilizes some of the dynamic attributes is mentioned above is provided as below for reference:

Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>
Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>
Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>
Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>
Prefix time: <http://www.w3.org/2006/time#>
Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>
Prefix xsd: <http://www.w3.org/2001/XMLSchema#>
Prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
select ?sensorID (max(?ti) as ?time) ?value ?latitude ?longitude
where {
    ?o a ssn:Observation.
    ?o ssn:observedBy ?sensorID.  
    ?o ssn:observedProperty ?qkr.
    ?qkr rdf:type ?qk.
    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}
    ?o ssn:observationSamplingTime ?t.
    ?o geo:location ?point.
    ?point geo:lat ?latitude.
    ?point geo:long ?longitude.
    ?t time:inXSDDateTime ?ti.
    ?o ssn:observationResult ?or.
    ?or  ssn:hasValue ?v.
    ?v dul:hasDataValue ?value.
    FILTER ((xsd:double(?latitude) >= "-90"^^xsd:double) &&   (xsd:double(?latitude) <= "90"^^xsd:double)
&& (xsd:double(?longitude) >= "-180"^^xsd:double) 
&& (xsd:double(?longitude) <= "180"^^xsd:double)) 
    FILTER(?value>="50"^^xsd:double)FILTER(?ti > "%%fromDateTime%%"^^xsd:dateTime && ?ti < "%%toDateTime%%"^^xsd:dateTime)
} group by ?sensorID ?time ?value ?latitude ?longi
tude

On top of the FISMO there is a FEMO object. Currently EEE supports the following FEMO template.

<fed:FEMO name="#FEMONAME#">
    <fed:description>#FEMODESCRIPTION#</fed:description>
    <fed:domainOfInterest>#DOMAINOFINTERESTLIST#</fed:domainOfInterest>
    <fed:FISMO name="#NAME#">
    …
    </fed:FISMO>
</fed:
FEMO>

Here:

  • #FEMONAME#: should be the name of the FEMO that experimenter want to have. For example:

<fed:FEMO name="MySecondExperiment">

  • #FEMODESCRIPTION#: should be the description of the FEMO to understand what is the FEMO is about. For example:

<fed:description>LargeScale crowdsensing experiment </fed:description>

  • #DOMAINOFINTERESTLIST#:  this is the list of the domain of interests that experiment supports. For multiple domain of interests values should be blank space separated. For example.

<fed:domainOfInterest>http://purl.org/iot/vocab/m3-lite#Transportation http://purl.org/iot/vocab/m3-lite#Pollution http://purl.org/iot/vocab/m3-lite#City http://purl.org/iot/vocab/m3-lite#Health</fed:domainOfInterest>

 

Further on the top of the FEMO object, there exists a FEDSpec object.

<fed:FEDSpec xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"            xmlns:fed="http://www.fiesta-iot.eu/fedspec"             xmlns:prt="http://www.w3.org/2007/SPARQL/protocol-types#"               xmlns:vbr="http://www.w3.org/2007/SPARQL/results#"               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"            xsi:schemaLocation="FILELOCATION" userID="#USERID#">
<fed:FEMO name="#FEMONAME#">
    …
</fed:FEMO>
</fed:FEDSpec>

One important thing to note here is the #USERID#. Please use your #USERID#. Do not use any other #USERID# in case you use it, you will not be able to view or perform any operations on the services you need.

For complete valid sample please consult the following table:

Experiment Instance (FEDSpec Example)

<?xml version="1.0" encoding="UTF-8"?>

<fed:FEDSpec xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"

 xmlns:fed="http://www.fiesta-iot.eu/fedspec"

 xmlns:prt="http://www.w3.org/2007/SPARQL/protocol-types#"

 xmlns:vbr="http://www.w3.org/2007/SPARQL/results#"

 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xsi:schemaLocation="http://www.fiesta-iot.eu/fedspec file:/C:/Ext_SSD/AIT/FIESTA/FIESTA-SVN/WP4/Task%204.1/Objects/XSD/FEDSpec.xsd" userID="YOURUSERID">

    <fed:FEMO name="MySecondExperiment">

        <fed:description>LargeScale crowdsensing experiment</fed:description>

        <fed:domainOfInterest>http://purl.org/iot/vocab/m3-lite#Transportation http://purl.org/iot/vocab/m3-lite#Pollution http://purl.org/iot/vocab/m3-lite#City http://purl.org/iot/vocab/m3-lite#Health</fed:domainOfInterest>

        <fed:FISMO name="2ndUseCase">

            <fed:description>Over time all noise observations for a given location</fed:description>

            <fed:discoverable>true</fed:discoverable>

            <fed:experimentControl>

                <fed:scheduling>

                    <fed:startTime>2016-11-08T18:50:00.0Z</fed:startTime>

                    <fed:Periodicity>250</fed:Periodicity>

                    <fed:stopTime>2017-11-08T18:49:59.0Z</fed:stopTime>

                </fed:scheduling>

            </fed:experimentControl>

            <fed:experimentOutput location="http://ExperimentServer.org/store/"></fed:experimentOutput>

            <fed:queryControl>

                <prt:query-request>

                    <query><![CDATA[

                        # [1 / 1] visualization type: 'Gauge' and sensors

                        Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

                        Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

                        Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

                        Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

                        Prefix time: <http://www.w3.org/2006/time#>

                        Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

                        Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

                        select ?s ?tim ?val

                        where {

                            ?o a ssn:Observation.

                            ?o ssn:observedBy ?s.  

                            ?o ssn:observedProperty ?qkr.

                            ?qkr a ?qk.

                            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                            ?o ssn:observationSamplingTime ?t.

                            ?o geo:location ?point.

                            ?point geo:lat "4.346104E1"^^xsd:double.

                            ?point geo:long "-3.80649E0"^^xsd:double.

                            ?t time:inXSDDateTime ?ti.

                            ?o ssn:observationResult ?or.

                            ?or  ssn:hasValue ?v.

                            ?v dul:hasDataValue ?val. 

                        } group by (?s) ?tim ?val

                    ]]></query>

                </prt:query-request>

              </fed:queryControl>

        </fed:FISMO>

        <fed:FISMO name="3rdUseCase">

            <fed:description>Over time noise observations for a given bounding box (time period in scheduling)</fed:description>

            <fed:discoverable>true</fed:discoverable>

            <fed:experimentControl>

                <fed:scheduling>

                    <fed:startTime>2016-11-08T18:50:00.0Z</fed:startTime>

                    <fed:Periodicity>250</fed:Periodicity>

                    <fed:stopTime>2017-11-08T18:49:59.0Z</fed:stopTime>

                </fed:scheduling>

            </fed:experimentControl>

            <fed:experimentOutput location="http://ExperimentServer.org/store/"></fed:experimentOutput>

            <fed:queryControl>

                <prt:query-request>

                    <query><![CDATA[

                        # [1 / 1] visualization type: 'Gauge' and sensors

                        Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

                        Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

                        Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

                        Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

                        Prefix time: <http://www.w3.org/2006/time#>

                        Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

                        Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

                        select ?s (max(?ti) as ?tim) ?val ?lat ?long

                        where {

                            ?o a ssn:Observation.

                            ?o ssn:observedBy ?s.  

                            ?o ssn:observedProperty ?qkr.

                            ?qkr a ?qk.

                            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                            ?o ssn:observationSamplingTime ?t.

                            ?o geo:location ?point.

                            ?point geo:lat ?lat.

                            ?point geo:long ?long.

                            ?t time:inXSDDateTime ?ti.

                            ?o ssn:observationResult ?or.

                            ?or  ssn:hasValue ?v.

                            ?v dul:hasDataValue ?val.

                            {

                                select  (max(?dt)as ?ti) ?s        

                                where {

                                    ?o a ssn:Observation.

                                    ?o ssn:observedBy ?s.  

                                    ?o ssn:observedProperty ?qkr.

                                    ?qkr a ?qk.

                                    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                                    ?o ssn:observationSamplingTime ?t.

                                    ?t time:inXSDDateTime ?dt.

                                }group by (?s)

                            }

                            FILTER (

                               (xsd:double(?lat) >= "-90"^^xsd:double)

                            && (xsd:double(?lat) <= "90"^^xsd:double)

                            && ( xsd:double(?long) >= "-180"^^xsd:double) 

                            && ( xsd:double(?long) <= "180"^^xsd:double)

                            ) 

                        } group by (?s) ?tim ?val ?lat ?long

                    ]]></query>

                </prt:query-request>

              </fed:queryControl>

        </fed:FISMO>

        <fed:FISMO name="4thUseCase">

            <fed:description>3rd usecase with noise more than x dB(A)</fed:description>

            <fed:discoverable>true</fed:discoverable>

            <fed:experimentControl>

                <fed:scheduling>

                    <fed:startTime>2016-11-08T18:50:00.0Z</fed:startTime>

                    <fed:Periodicity>250</fed:Periodicity>

                    <fed:stopTime>2017-11-08T18:49:59.0Z</fed:stopTime>

                </fed:scheduling>

            </fed:experimentControl>

            <fed:experimentOutput location="http://ExperimentServer.org/store/"></fed:experimentOutput>

            <fed:queryControl>

                <prt:query-request>

                    <query><![CDATA[

                        # [1 / 1] visualization type: 'Gauge' and sensors

                        Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

                        Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

                        Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

                        Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

                        Prefix time: <http://www.w3.org/2006/time#>

                        Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

                        Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

                        select ?s (max(?ti) as ?tim) ?val ?lat ?long

                        where {

                            ?o a ssn:Observation.

                            ?o ssn:observedBy ?s.  

                            ?o ssn:observedProperty ?qkr.

                            ?qkr a ?qk.

                            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                            ?o ssn:observationSamplingTime ?t.

                            ?o geo:location ?point.

                            ?point geo:lat ?lat.

                            ?point geo:long ?long.

                            ?t time:inXSDDateTime ?ti.

                            ?o ssn:observationResult ?or.

                            ?or  ssn:hasValue ?v.

                            ?v dul:hasDataValue ?val.

                            {

                                select  (max(?dt)as ?ti) ?s         

                                where {

                                    ?o a ssn:Observation.

                                    ?o ssn:observedBy ?s.  

                                    ?o ssn:observedProperty ?qkr.

                                    ?qkr a ?qk.

                                    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                                    ?o ssn:observationSamplingTime ?t.

                                    ?t time:inXSDDateTime ?dt.

                                }group by (?s)

                            }

                            FILTER (

                               (xsd:double(?lat) >= "-90"^^xsd:double)

                            && (xsd:double(?lat) <= "90"^^xsd:double)

                            && ( xsd:double(?long) >= "-180"^^xsd:double) 

                            && ( xsd:double(?long) <= "180"^^xsd:double)

                            )  

                            FILTER(?val>="75"^^xsd:double)

                        } group by (?s) ?tim ?val ?lat ?long

                    ]]></query>

                </prt:query-request>

              </fed:queryControl>

        </fed:FISMO>

        <fed:FISMO name="5thUseCase">

            <fed:description>3rd usecase with noise less than x dB(A)</fed:description>

            <fed:discoverable>true</fed:discoverable>

            <fed:experimentControl>

                <fed:scheduling>

                    <fed:startTime>2016-11-08T18:50:00.0Z</fed:startTime>

                    <fed:Periodicity>250</fed:Periodicity>

                    <fed:stopTime>2017-11-08T18:49:59.0Z</fed:stopTime>

                </fed:scheduling>

            </fed:experimentControl>

            <fed:experimentOutput location="http://ExperimentServer.org/store/"></fed:experimentOutput>

            <fed:queryControl>

                <prt:query-request>

                    <query><![CDATA[

                        # [1 / 1] visualization type: 'Gauge' and sensors

                        Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

                        Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

                        Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

                        Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

                        Prefix time: <http://www.w3.org/2006/time#>

                        Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

                        Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

                        select ?s (max(?ti) as ?tim) ?val ?lat ?long

                        where {

                            ?o a ssn:Observation.

                            ?o ssn:observedBy ?s.  

                            ?o ssn:observedProperty ?qkr.

                            ?qkr a ?qk.

                            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                            ?o ssn:observationSamplingTime ?t.

                            ?o geo:location ?point.

                            ?point geo:lat ?lat.

                            ?point geo:long ?long.

                            ?t time:inXSDDateTime ?ti.

                            ?o ssn:observationResult ?or.

                            ?or  ssn:hasValue ?v.

                            ?v dul:hasDataValue ?val.

                            {

                                select  (max(?dt)as ?ti) ?s        

                                where {

                                    ?o a ssn:Observation.

                                    ?o ssn:observedBy ?s.  

                                    ?o ssn:observedProperty ?qkr.

                                    ?qkr a ?qk.

                                    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient}

                                    ?o ssn:observationSamplingTime ?t.

                                    ?t time:inXSDDateTime ?dt.

                                }group by (?s)

                            }

                            FILTER (

                               (xsd:double(?lat) >= "-90"^^xsd:double)

                            && (xsd:double(?lat) <= "90"^^xsd:double)

                            && ( xsd:double(?long) >= "-180"^^xsd:double) 

                            && ( xsd:double(?long) <= "180"^^xsd:double)

                            )  

                            FILTER(?val<="45"^^xsd:double)

                        } group by (?s) ?tim ?val ?lat ?long

                    ]]></query>

                </prt:query-request>

</fed:queryControl>

        </fed:FISMO>

    </fed:FEMO>

</fed:FEDSpec>

1.3 Best practices

Creating the FEDSpec

To consolidate the previous section, we encourage the experimenters to follow the best practices. These include:

  • Please do not use the template/provided FEDSpec’s as it is. The provided template is a generic template. We further encourage you to remove all #*# and with it the properties that are not used. For the properties you are using, change #*# with the actual values you want to use.  The data format of the values you provide should be same as that reflected in the examples. Further, use logical values. Please do not use Test, Blah Blah, etc, in the description and other places.
  • Have userID specified in the FEDSpec. You should modify #USERID#. Your FEDSpec will not be saved if #USERID# is different from the ACCESSTOKEN used.
  • fed:startTime should be less than fed:stopTime. Ideally, the difference between startTime and stopTime should not be 1 sec. If this is the case, this will only allow the experiment to be executed once in the lifetime. If this is a needed functionality we recommend to do so. However there is another option. Instead of scheduling the experiment on the Execution Engine, experimenters can use the Polling option available via Management Console. To avail Polling service, experimenters will first have to deploy the FEDSpec, but are not supposed to “START” the execution of a particular FISMO. By not starting the execution, the FISMO status remains “NOT YET SCHEDULED”. If only one execution is not the needed functionality please be realistic in the difference. Do not set #PERIODICITY# to 1 second for the queries. This will only overload your system, block the network without providing any insight. As a best practice we recommend you to use periodicity of not less than 600.
  • If you use dynamic query parameters please use them as specified for the query, i.e., using %%fromDateTime%%, %%toDateTime%%, %%geoLatitude%%, and %%geoLongitude%%. For other cases, other parameters used in dynamic query, use %%DA_NAME%%.
  • Before writing the queries we strongly encourage you to have a look at the documentation of the kind of data is currently provided and what is the semantic structure. We recommend you to write queries that would serve your purpose and are efficient. Please DO NOT:
    • Write queries that are irrelevant for your experiment. This will have overloading effect on your Experiment.
    • Use overly complex queries for simple cases unless no other solution.

We would be happy to help you write efficient queries. However, as an experimenter we expect that you first write queries and we validate. Additionally do not use “Select * Where {?s ?p ?o.}” query, this goes against our principles and shall be completely prohibited. More details on the best practices for efficient querying is provided in the next section. Further, in case of specific queries, if you have something with “PARAMETER”, we recommend you to use \”PARAMETER\”. Note the \”. This is done to correctly parse the query when invoking IoT-Registry APIs.

  • Note that the Execution Engine uses UTC timezone. Thus all your queries should be using UTC timezone. Moreover, the fed:startTime and fed:stopTime should also be specified in UTC timezone.
  • Sync the #PERIODICITY# and the used time interval in the query. If not synchronized, it is possible that you get redundant data that is not useful. Generating redundant data will again block resources and will be not efficient.
  • Please test the #URLLOCATION# before providing it. Please make sure that the URL is operational and accepts incoming data. To help experimenters, a sample code is provided that opens a URL that accepts incoming data and stores it in a file. Note that, as EEE is operating in an ASYNC mode, the result when available is sent to the experimenter. An experimenter should periodically check if the data is available on their server or not. A very simple solution to not write a periodic checker is to get the file that is most recently created. All the files that are sent are time-stamped. The naming convention followed to name the such files is 

String filename=JOBID.replace(“-”,””)+URLLOCATION.replace(":", "").replace("/", "_")+”_”+LONG_TIMESTAMP;

Note here JOBID is a UUID that is set by EEE, URLLOCATION is the location that you provide and LONG_TIMESTAMP is milliseconds after epoch.

  • In case you are executing the experiment using FEDSpec, you need to install Experiment Data receiver module on your server. This module will enable a valid #URLLOCATION# that can be utilized in the FEDSpec. The Experiment Data receiver installation guide is available at the following link

https://github.com/fiesta-iot/experiment.data.receiver/blob/master/ExperimentServer/Readme.md

Please note that the module is currently tested on Tomcat and creates a #URLLOCATION# that looks like http(s)://HOST:PORT/ExperimentServer/store. In case you are using HTTPS please make sure that you are using LetsEncrypt/JVM Already trusted Certificates/Terena SSL root certificate. In case any other certificate is used the Experiment Execution Engine will not be able to send the resultset to #URLLOCATION#. Self-signed certificates will not work. In case you use HTTP, there is no such restriction. The priority is given to the link that is specified. In case this URL is not reachable for some reason, the results are stored locally within FIESTA-IoT repositories for experimenters to later download the results. Note that the experimenters have to create their own java code to download the results. The complete description of the API that provides this functionality is described later

In case you decide to directly call the IoT-Registry APIs, you need not install this component. Nevertheless, you will have to deal with authenticating yourself every time you are calling IoT-Registry APIs, scheduling your experiment with in SYNC periodicity and time interval used in the query (Still you need to follow #3 above).

  • In the fed:presentationAttr value of the widget attribute please follow the guidelines as said in the section above. The value should be a JSON string that is of form:

{
  &quot;Method&quot;: [&quot;fft&quot;&quot;linReg&quot;],
  &quot;Parameters&quot;: [&quot; &quot;, &quot;Predict&quot;]
}

  • Please send us your FEDSpec and the queries (for advanced experimenters) that you intend to execute, get it validated before you proceed with registering the FEDSpec into the system or using the query and directly calling the IoT-Registry API. In case you want to test your queries, before providing them (in the FEDSpec and scheduling them, giving it to us to validate) you can still yourself validate the queries. Note that here ve refer to just getting a feel of what data is available and if the syntax of the query is correct. Based on our recommendations about your experiments you need to decide if creating FEDSpec is a best option of directly using IoT-Registry is best for you.

Note that there are 2 aspects that we mentioned before validating the syntax and getting a feel of the data. You can validate the syntax of the query using http://www.sparql.org/query-validator.html. This will ensure that your query is executable on Fiesta−IoT Platform. To get a feel of what the query will return you can use any REST Client (Postman, Google Chrome’s Advanced REST Client, Curl, etc.) to send your request to Fiesta−IoT Platform. There are 3 main APIs that you need to deal with:

The Authenticate API will give you an access token that you will have to use to call the other 2 APIs. Using these APIs you can execute your query and test them. Note that the APIs used should be based on which graph you want to query.



Writing efficient Queries

We list below the best practices to query the system. Further we also ask you to consult the following tables to know what kind of data is currently present in FIESTA-IoT repository:

SmartSantander

SmartICS

KETIs Mobius

SoundCity

○      If you run first a resource discovery, you can harness sensor information to save much time in further queries (i.e. observations based on node ids).

○      If your experiment aims at short-term data (not historical values), another thing that can save time is the usage of IoT-Service endpoints instead of raw SPARQLs queries. However, please note this concept is not available for all the sensors. It can be used wherever provided. In the ontology it is represented via iot‑lite:Service.

○      In case you do not need the complete graph structure, please do not query for all the concepts and properties.

○      The entire structure is divided into 2 graphs: Resource graph and observation graph. The queries must be directed to either of these graphs based on the requirements. Query these graphs is resource efficient. In case both graphs are to be queried, you must use another graph called “global”. We recommend you to not query this graphs unless essential.

○      Learn and understand the meaning of the FILTER, GROUP, LIMIT, BIND etc. options, and try to use them if possible. Note that adding such keywords slows down the query execution. For example, bind significantly degrades the execution time.

○      Try, to the extent possible, to avoid the extraction of large amounts of data at once (e.g. >5MB). In this case, split your queries into various ones; for instance, sweeping the time into small windows, etc.

○      There is a possibility that the dul:dataValue returned is a NaN. This NaN is mainly reported currently by the SmartICS testbed. Thus, it is useful, in case you do not want to receive observations that have NaN value to filter such observations. In FIESTA-IoT, currently some observations have dul:dataValue as NaN while some have dul:dataValue as NaN^^xsd:string. Note the absence of data-type in the first case.

            A filter command looks like

            FILTER (?dataValue != NaN^^xsd:string || ?dataValue != NaN)

The dul:dataValue currently can return following datatypes xsd:int, xsd:double, xsd:dateTime, xsd:boolean, xsd:string. Thus it is of utmost importance that experimenters look into what kind of data they need and understand the mappings between QuantityKinds, Units and datatypes.

○      Each sensor/resource has EXACTLY one QuantityKind and Unit associated to it. Please refer to Testbed documentation (link below) to understand what phenomenon is being mapped to which m3-lite concept.

■      Some other relevant documents are available at:

○      IoT-Service endpoints are a good deal when it comes to get the last observations carried out by the sensors. However, there is a number of points that has to be considered beforehand;

■      FIESTA-IoT does not specify the format of the response messages (in the current version of the platform). This means that every testbed might follow a different data set. Thus, experimenters have to manually parse them.

■      It is worth highlighting again that these services are not mandatory for testbeds, so they might (or might not) decide to include these endpoints as part of the resource description. Indeed, up to today (1st Aug 2017), 2 out of 4 testbeds in the federation (i.e. SmartSantander and SmartICS) do offer this possibility.

■      Even though there is a kind of de-facto agreement on the actual use of the endpoints, that is, to expose the last measurement observed by a node, this is not an official standard. Consequently, testbed providers might use the endpoints for a different purpose.

FIESTA-IoT testbeds are deployed (SoundCity testbed is actually a crowdsensing platform and is not bound to a particular physical location) on different geographical areas: i.e. Spain (SmartSantander), UK (SmartICS) and South Korea (KETI). Nevertheless, SoundCity testbed does not have any specific location as this testbed uses crowdsourced information coming from uses located in different locations around the globe. Furthermore, more than 6 testbeds will gradually the federation.

1.4 Register a new experiment

FIESTA-IoT is currently offering a simple interface in order to Store, Update and delete experiments called Experiment Register Client. The Experiment Register Client can be found at the Experimenter menu of the FIESTA-IoT portal (see 1st Figure below).

The Experiment Register Client provides the ability to store an experiment at the FIESTA−IoT platform in the form of a FEDSpec. The defined FEDSpec could be as simple as a single service (FISMO) or as complex as multiple experiments (FEMOs). To upload a FEDSpec first one should identify the location of it by hitting the “Open FEDSpec” (see 2nd Figure below) and then by hitting the “Save FEDSpec” button. As soon as the FEDSpec is saved the included FEMOS appears in the available experiments list (FEMOS) as shown in 2nd Figure below. When uploading a FEDSpec the FEMO/FISMO IDs should be empty, as they will be automatically assigned by the system. 

By choosing, a FEMO from the User is capable to have a quick overview of it as shown in 3rd Figure below.

The tools provides also the ability to export a FEMO by hitting the “Export FEDSPEC” button after choosing the FEMO of interest from the provided list. The FEDSpec that will be exported will now contain the FEMO/FISMO IDs assigned from the FIESTA−IoT platform. This will give the Experimenter the ability to update the exported FEMO/FISMO by updating the XML file and saving it again to the Experiment Repository following the same process described above.

1.5 Run/Execute an experiment

An experiment can be executed in many ways and FIESTA-IoT provides solutions for the execution of experiment for two categories of users (novice, advanced). Further, for the advanced user case FIESTA-IoT provides 2 solutions: one based on APIs of “Experiment execution Engine” (EEE) and another based on directly accessing IoT-Registry APIs. Novice experimenters are advised to use the method described in Section "Run/Execute an experiment (Novice Experimenter)".

As said experiment execution is handled by a component called “Experiment execution Engine” or EEE. This module uses and supports the experiment description written by an experimenter in the domain specific language (DSL) format specified by FIESTA-IoT (for reference on the DSL refer to Section 1.1 "Experiment DSL (FEDSpec)"). Amongst the available features in the DSL, in the current version, EEE supports only a few.  These include starting an experiment service (FISMO), pausing a FISMO, restarting a FISMO, subscribing to already existing and discoverable FISMOs, unsubscribing from subscribed FISMOs, and polling a FISMO (executing one time a FISMO on the Fiesta−IoT Platform).  The EEE specific APIs are available here for developers or experimenters for testing and more in-depth knowledge about specific APIs. Note that in case an experimenter wants to use the EEE API they should still upload the FEDSpec either using the ERM API (Section 0) or the ERM Client (Section 1.4). Nevertheless, experimenters can also use Experiment Management Console and perform actions on the FISMO. This option is to be used by novice experimenters.



Run/Execute an experiment (Novice Experimenter)

In order to execute an experiment using Experiment Management Console that is described by its FISMOs, the Experimenter first need to go to: https://platform.fiesta-iot.eu/experimentConsole/experimentConsole.jsp.

You can also use the cookie version of the console by just using the link above. Upon successful authentication list of experiments associated with the experimenter or the user is retrieved as shown in following figure. Note that this is also available via portal. The experimenter needs to go to the Experimenter Menu and click the “Experiment Management Console”.





From this view, experimenters can then select whichever experiment they want to work on from the list using the “SELECT” button next to each experiment.  Once a particular experiment is selected, this would open another UI  (as shown in 2nd Figure to 4th Figure, the entire UI is divided into 3 panes: Experiment Details, Associated FISMOs, and Subscription Pane) where experiment name, experiment description, a list of experiment Domain of Interest along with Associated FISMOs and available FISMOs for subscription is shown.

An eexperimenter can choose to update the metadata of the experiment that he/she has created using “EXPERIMENT EDITOR”. This will open the UI provided in Section Define an experiment through Domain Specification Language (DSL) where experimenter can resubmit their updated Experiments. Upon these resubmissions, a service in EEE is triggered that changes only the scheduling interval. If the scheduling interval is not changed nothing is updated on the EEE.

The “Associated FISMOs” pane shows the “meta” information about the FISMOs that are associated to a particular experiment. The “meta” information includes: if the FISMO was Owned or Subscribed within the frame of an experiment, the status of the FISMO (either NOT YET SCHEDULLED, NORMAL, PAUSED, etc.), past execution history and polling option. Once scheduled a “delete job” button will appear that will let experimenters delete any reference of a particular FISMO from EEE. Upon deleting the FISMO will not be executed any more. In order to check for the description of the FISMO, experimenter can click on the name. This will open a snackbar in the bottom of the page and will show the description of the selected FISMO.

Initially, all the FISMOs have the NOT YET SCHEDULLED status. If the experimenter wants to start the FISMO, they can switch the toggle button. Upon first toggle, the FISMO will be scheduled by the EEE with the NORMAL status. Another toggle would PAUSE the FISMO execution. Yet another toggle would restart the PAUSED FISMO. In order to successfully schedule the FISMO execution, the current version of the EEE supports all that is specified in Section "Define an experiment through Domain Specification Language (DSL)".

A sample of <fed:scheduling> is provided below:

<

fed:scheduling>

       <fed:startTime>2016-11-08T18:13:51.0Z</fed:startTime>

      <fed:Periodicity>600</fed:Periodicity>

      <fed:stopTime>2017-11-08T18:13:50.0Z</fed:stopTime>

</fed:scheduling>



The <fed:scheduling> would provide the EEE with the start date, end date and the periodicity of the FISMO execution. Thus making these attributes essential in the FISMO description. Once the schedule is set in the EEE, EEE provides a JOB ID that is used for internal purposes. This JOB ID is then provided with the status NORMAL. Upon the schedule, the <query> is read by the EEE from the FISMO description and is sent to FIESTA-IoT Meta-Cloud.

The Meta-Cloud executes the query and sends back the results to the EEE. The EEE stores the result internally and pings the location specified in the location specified by the <fed:experimentOutput> (<fed:experimentOutput location=“location”/>).  Upon success, the results are sent to the specified location and deleted from the internal repository. Currently, EEE assumes that the “location” here is a URL, where the specified credentials are granted to the EEE to write the results in a file. For reference and ease, a sample code that experimenters can execute on their server can be found in the following public repository.

It is thus noteworthy to state that currently EEE only supports one mechanism right now to send the information to the experimenter. Given the above, it is thus essential to specify <fed:scheduling> <experimentControl> attribute of FISMO, <query> under <prt:query-request> under <fed:queryControl> and <fed:experimentOutput  location=“location”>. 

If If the experimenter wants to just execute the FISMO and not wait for the EEE to trigger the execution of the FISMO, the experimenter can use POLL NOW. The POLL NOW will execute the <query> defined within the FISMO and would return the results to the same URL that is specified (i.e. the URL where results of scheduled execution are being sent).

Nonetheless, apart from the above functionality, an experimenter can also subscribe to an already existing FISMO (If there were no FISMOs available for subscription this part would be disabled). In case there are many FISMOs available, an experimenter can choose a particular FISMO from the dropdown list and provide the URL information (see 4th Figure). Note that as EEE only support URL, experimenters must specify a valid endpoint. Only after validating the experimenter’s URL the “SUBSCRIBE” button will be unlocked. The experimenter can currently only choose one FISMO at a time.

Once successfully subscribed, the list of Associated FISMOs is updated to show the subscriptions. Each new subscription would provide a new JOB ID where the status of the JOB would be NORMAL to the subscribed FISMO and its execution would begin as the schedule specified in the description of that particular subscribed FISMO (see 5th Figure).

Moreover, the URL specified in the FISMO will not be used. Instead, the URL specified by the subscriber would be used to forward the results. An experimenter, if wishes, can unsubscribe the subscribed FISMO by clicking “UNSUBSCRIBE”. This will delete the JOB associated from the EEE. Note that in the 5th Figure, the Name of 2 FISMOs is same. This can occur due to the fact that another Experimenter has named one of his FISMO exactly the same.

An experimenter is also given a capability to see the details of past executions of the “Associated FISMOs”. The details are provides in the form of a graph and contains information like how much time did it take to execute the FISMO and how much data was obtained from the Meta-Cloud. This graph however not show how much time did it take to execute the FISMO and how much data was obtained from the Meta-Cloud when the FISMO is polled.

In order to delete the experiment it is advised that experimenters first stop/delete the execution of any related FISMO objects on the EEE using the Experiment Management Console. Once this is done, they are advised to remove the experiment from the Experiment Registry Client. We acknowledge this workflow because this will give experimenters a view of what all services are running and if it is really required to remove them at all.



Run/Execute an experiment (Advanced Experimenter)

This method is an API based method. The workflow to model the execution of the experiment should essentially follow:

  • Use ERM API to Store the experiment
  • Use EEE API to schedule, poll, subscribe or monitor the experiment.
  • To update FEDSpec follow the same process as described above (delete the execution first)

Further, to help experimenters use the dynamic feature, we also provide an API. This API should only be used by advanced experimenters and thus is not available via Experiment Management Console. Advance experimenters will have to use the API to execute their dynamic queries. It is important to note that experimenters can use ERM Client to store the FEDSpec and then use the EEE APIs to execute their experiments. Nevertheless, the API for dynamic queries has a signature as described below:

API: /polling/dynamicPollForReport

Method: POST

Produces: application/json

Header Parameters:

  • fismoID: required, fismoID of the service. It is in string format
  • femoID: required, femoID of the experiment. It is in string format
  • iPlanetDirectoryPro: required, it is the access token. It is in string format

Query Parameters:

  • owner: default value is true, this is to identify if the fismo was subscribed or not
  • geoLatitude: default value is 0. However, experimenters need to set this as a double value but in a string format
  • geoLongitude: default value is 0. However, experimenters need to set this as a double value but in a string format
  • intervalNowToPast: time in millisecond. The format is int. Default value is 0
  • fromTime: it is a long value to represent milliseconds after epoch. Default value is 0
  • toTime: it is a long value to represent milliseconds after epoch. Default value is 0

Body Parameters:

  • This is a JSON object represented as a string. The default value is “{}”. However, experimenters need to set the key value pair depending on the query. A JSON object the experimenters need to set is

{
    "KATInput":{"Method":[""], "Parameters":[""]},
    "otherParameters":{<key>:<value>}
}

Here, KATInput reflects the input needed for the FIESTA-IoT Analytics Toolkit, while otherParameters reflect the dynamic attributes.

Response:

  • 200: A successful message looks like {"response":"Polled successfully", "jobID": <JOBID>}
  • 400: Some of the 400 responses include: {"response":"PersistenceException"}, {"response":"ImplementationException"} and error message in JSON format coming from ERM module.

Nevertheless, advanced experimenters can also create their own EEE like component and directly interact with the IoT-Registry. In this section we refrain ourselves in describing the IoT-Registry APIs. These APIs are described in Section 5.3.



Experiment Result Sending and Storage

Having executed the experiment via EEE, it is also essential that we describe the method for sending data to the experimenters. Currently there are many ways experimenters can get the result they intend. These are:

  • Install Experiment Data receiver and provide URLLOCATION for the /store API in the FEDSpec. The process to install and provided as a readme (See here).
  • In case your URLLOCATION was down for some reason, the EEE stores the un-send resultsets locally. The un-send reseultset can be downloaded by the Experimenter.
  • In case you are advance experimenter and not using EEE, you need to setup infrastructure to receive data from IoT-Registry. Remember, as the execution of the query might take time, you need to set Request Timeout property accordingly to a large number.



Experiment Data receiver

A sample code is provided for experimenters to receive data provided by EEE. This code is tested to run on Tomcat successfully.

To deploy, on the experimenter side following has to be done before the deployment

  • in the web.xml change the location entry for multipart-config with the desired location

<multipart-config>  <location>#LOCATION#</location>  </multipart-config>

  • In the 

src/eu/fiesta_iot/experimentServer/ExperimentServerService.java change the $(LOCATION) to match the location set in the web.xml

File file = new File("${LOCATION}", fileName)

Note that this location is the desired location where you want to store the received files.

  • Make sure that the #LOCATION# has read-write permissions to the Tomcat user and group (under the name and group Tomcat is running).
  • In the Tomcat server change the following line in the conf/content.xml

 <Context>      ...  </Context>

with the following

 <Context allowCasualMultipartParsing="true">      ...  </Context>

  • restart Tomcat server

Once the above is done, do the following

  • cd <PATH TO EXPERIMENTSERVER>
  • mvn clean install
  • cp <PATH TO EXPERIMENTSERVER>/target/ExperimentServer.war <PATH TO TOMCAT WEBAPPS>

 

Your service is running at http(s)://<HOST>:<PORT>/ExperimentServer/store/ and thus your URLLOCATION should be http(s)://<HOST>:<PORT>/ExperimentServer/store/

This will enable you to receive the resultsets that are generated after the execution of your FISMO. As also stated before the name of the file received follows a naming convention. Again it is:

String filename = JOBID.replace(“-”,””)+URLLOCATION.replace(":", "").replace("/", "_")+”_”+LONG_TIMESTAMP;

Note here JOBID is a UUID, URLLOCATION is the location that you provide and LONG_TIMESTAMP is a Time-Stamp in long (milliseconds after epoch).

If the experimenter is using HTTPS, then they should use LetsEncrypt certificate for this API/URL. All other certificates other than those available in default JVM configuration will fail. This is because of the JVM does not have all the certificates installed. However, if the URL is HTTP, it will pass through.



Experiment Result Storage

All resultsets that cannot be sent to the experimenters are stored in the Experiment Result Storage (ERS). ERS stores resultset as is and returns them when service is invoked. Upon a success the particular resultset is deleted from the store.

Experimenters need to use an ERS API to download the needed data. This API has a signature:

API: /experiment-result-store
Method: GET
Accepts: application/json
Produces: application/json

Header Parameters:

  • iPlanetDirectoryPro: required, it is the access token. It is in string format
  • femoID: required, femoID of the experiment. It is in string format
  • jobID: optional, JobID of the FISMO in the EEE

If both FEMOID and JobID are provided, then the corresponding FISMO results are returned.

If only the FEMOID is provided, then all FISMO execution results under that particular FEMO along with its corresponding job IDs are returned.

Response:

  • 200: A successful message contains following template:

{ "femoResults": [

   {   "jobid": "<JOBID>",

       "results": [                

               { "time": "<TIMESTAMP>",

             "result": "<RESULTSET>"

           }

        ]

     },……

]}

A sample looks like  

{ "femoResults": [

{

iot/resource/sc-sics-sp-002-power,http://smart-ics.ee.surrey.ac.uk/fiesta- iot/resource/sc-sics-sp-001-power\n0.0,1.432187701766452e- 14,1.9040324872321435e- 14\n0.00819672131148,10.483904864244515,12.66849135485158\n0.016393442623, 10.037536235817262,11.694920662095793\n0.0245901639344,8.274362461842944,1 0.787303593017295\n0.0327868852459,6.936821514862263,8.95507559616293\n0.0 409836065574,5.322912117131182,6.684190365119811\n0.0491803278689,4.082843 4819081725,4.156983856615794\n0.0573770491803,2.267728032490481,2.53056132 0212919\n0.0655737704918,0.6565507982225585,0.6050609198606894\n0.07377049 18033,1.3997255070742531,1.0799031869904239\n0.0819672131148,1.66570166045 3452,2.954284570154875\n0.0901639344262,1.6320456294790227,2.5005243782136 066\n0.0983606557377,1.5195656197778484,2.7019058846634727\n"
}
]
}
]}

  • 400: Some of the 400 responses include:

{"response":"<APPROPRIATE RESPONSE MESSAGE>"}

2 ERM API for Advanced Experimenters

Experiment Registry Management (ERM) component exposes an API that facilitates the experiment storage, retrieval and discovery. This is achieved by manipulating objects that comply with the FEDSpec schema and its various defined entities.

2.1 API Definition

The 1st Table below illustrates the main API primitives that support the Experiment Registry Management functionalities, while 2nd Table below provides more details about each one of the functions that comprise the API.

List of primitives comprising the Experiment Registry Management API

<<interface>>

ExperimentRegistryManagementInterface

---

POST:saveUserExperiments(fedSpec:FEDSpec):String

POST:deleteUserExperiments (userID:String):String

POST:saveUserExperiment(femo:FEMO, userID:String):String

POST:deleteUserExperiment (femoID:String):String

POST:saveExperimentServiceModelObject (fismo:FISMO, femoID:String):String

POST:deleteExperimentServiceModelObject (fismoID:String):String

GET: getALLUserExperiments (userID:String):FEDSpec

GET:getAllUserExperimentsDescreptions (userID:String):ExpDescriptiveIDs

GET: getExperimentDescreption (femoID:String):FemoDescriptiveID

GET:getExperimentModelObject (femoID:String):FEMO

GET:getExperimentServiceModelObject (fismoID:String):FISMO

           



The FIESTA-IoT implements the methods of the Experiment Registry Management API as specified in the 2nd Table below:

Experiment Registry Management API definition

2.2 Object Definition

The FEDSpec XSD, which was described in detail in 1.1 above, can be found in the following notes more specifically in 1st Table below. In the 1st Figure below we can see the ExpDescriptiveIDs schema graph (the XSD is provided in the notes below too and more specifically in the 2nd Table) which consists of:

  • A list of FEMO Descriptive ID: which is capable of providing a high level deception of an experiment and It includes:
    • The ID of the experiment
    • The description of the experiment
    • The name of the experiment and
    • A list of FISMO Descriptive ID: which is capable of providing a high level deception of a Service Model Object and It includes:
      • The FISMO ID
      • The FISMO description and
      • The FISMO name

Notes: 

Experiment Schemata

 

 

FEDSpec Schema

<?xml version="1.0" encoding="UTF-8"?>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"

 elementFormDefault="qualified" targetNamespace="http://www.fiesta-iot.eu/fedspec"

 xmlns:fed="http://www.fiesta-iot.eu/fedspec" xmlns:prt="http://www.w3.org/2007/SPARQL/protocol-types#">

 

 <xs:import namespace="http://www.w3.org/2007/SPARQL/protocol-types#"

  schemaLocation="sparql/protocol-types.xsd" />

 

 <xs:element name="FEDSpec">

  <xs:annotation>

   <xs:documentation>FIESTA Experiment Description Specification

   </xs:documentation>

  </xs:annotation>

  <xs:complexType>

   <xs:sequence>

    <xs:element maxOccurs="unbounded" ref="fed:FEMO" />

   </xs:sequence>

   <xs:attribute name="userID" use="required" type="xs:anyURI" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="FEMO">

  <xs:annotation>

   <xs:documentation>FIESTA Experiment Model Object

   </xs:documentation>

  </xs:annotation>

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" maxOccurs="1" ref="fed:description" />

    <xs:element minOccurs="0" maxOccurs="1" ref="fed:EDM" />

    <xs:element ref="fed:domainOfInterest" />

    <xs:element maxOccurs="unbounded" ref="fed:FISMO" />

   </xs:sequence>

   <xs:attribute name="id" use="optional" type="xs:anyURI" />

   <xs:attribute name="name" type="xs:NCName" use="required" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="description" type="xs:string" />

 

 <xs:element name="EDM" type="xs:string">

  <xs:annotation>

   <xs:documentation>Experiment design metadata.</xs:documentation>

  </xs:annotation>

 </xs:element>

 

 <xs:element name="FISMO">

  <xs:annotation>

   <xs:documentation>FIESTA Service Model Object</xs:documentation>

  </xs:annotation>

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" maxOccurs="1" ref="fed:description" />

    <xs:element minOccurs="0" name="discoverable" type="xs:boolean"

     default="false" />

    <xs:element ref="fed:experimentControl" />

    <xs:element ref="fed:experimentOutput" />

    <xs:element ref="fed:queryControl" minOccurs="0" />

    <xs:element name="service" nillable="false" type="xs:anyURI"

     minOccurs="0" />

    <xs:element ref="fed:rule" minOccurs="0" />

   </xs:sequence>

   <xs:attribute name="id" use="optional" type="xs:anyURI" />

   <xs:attribute name="name" type="xs:NCName" use="required" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="queryControl">

  <xs:complexType>

   <xs:sequence>

    <xs:element ref="fed:quantityKind" minOccurs="0" />

    <xs:element ref="fed:staticLocation" minOccurs="0" />

    <xs:element ref="fed:queryInterval" minOccurs="0" />

    <xs:element ref="prt:query-request" maxOccurs="1"

     minOccurs="1" />

    <xs:element maxOccurs="1" minOccurs="0" ref="fed:dynamicAttrs" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="domainKnowledge" type="xs:anyURI" />

 

 <xs:element name="staticLocation">

  <xs:complexType>

   <xs:sequence>

    <xs:element name="latitude" type="xs:string" />

    <xs:element name="longitude" type="xs:string" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="queryInterval">

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" name="fromDateTime" type="xs:dateTime" />

    <xs:element minOccurs="0" name="toDateTime" type="xs:dateTime" />

    <xs:element minOccurs="0" name="intervalNowToPast" type="xs:int" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="experimentControl">

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" ref="fed:scheduling" maxOccurs="1" />

    <xs:element name="trigger" type="xs:anyURI" minOccurs="0" />

    <xs:element name="reportIfEmpty" type="xs:boolean"

     minOccurs="0" default="true" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="rule">

  <xs:complexType>

   <xs:sequence>

    <xs:element ref="fed:ruleDefinition" />

    <xs:element ref="fed:domainKnowledge" />

    <xs:element ref="fed:quantityKind" />

   </xs:sequence>

   <xs:attribute name="name" type="xs:string" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="ruleDefinition" type="xs:string">

  <xs:annotation>

   <xs:documentation>i.e. Jena rule or SPARQL construct

   </xs:documentation>

  </xs:annotation>

 </xs:element>

 

 <xs:element name="domainOfInterest">

  <xs:annotation>

   <xs:documentation>List of URLs linking with M3-lite taxonomy.

   </xs:documentation>

  </xs:annotation>

  <xs:simpleType>

   <xs:list itemType="xs:anyURI" />

  </xs:simpleType>

 </xs:element>

 

 <xs:element name="quantityKind">

  <xs:annotation>

   <xs:documentation>List of URLs linking with M3-lite taxonomy.

   </xs:documentation>

  </xs:annotation>

  <xs:simpleType>

   <xs:annotation>

    <xs:documentation>URL linking with M3-lite taxonomy.

    </xs:documentation>

   </xs:annotation>

   <xs:list itemType="xs:anyURI" />

  </xs:simpleType>

 </xs:element>

 

 <xs:element name="scheduling">

  <xs:complexType>

   <xs:all>

    <xs:element form="qualified" name="startTime" type="xs:dateTime"

     minOccurs="0" />

    <xs:element name="Periodicity" minOccurs="0" maxOccurs="1"

     type="xs:int" />

    <xs:element minOccurs="0" name="stopTime" type="xs:dateTime" />

   </xs:all>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="experimentOutput">

  <xs:complexType>

   <xs:sequence maxOccurs="1" minOccurs="1">

    <xs:element minOccurs="0" ref="fed:file" maxOccurs="unbounded" />

    <xs:element maxOccurs="unbounded" ref="fed:widget"

     minOccurs="0" />

   </xs:sequence>

   <xs:attribute name="location" type="xs:anyURI" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="file">

  <xs:complexType>

   <xs:sequence>

    <xs:element name="type" type="xs:string" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="widget">

  <xs:complexType>

   <xs:sequence>

    <xs:element maxOccurs="unbounded" ref="fed:presentationAttr" />

   </xs:sequence>

   <xs:attribute name="widgetID" use="required" type="xs:anyURI" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="presentationAttr">

  <xs:complexType>

   <xs:attribute name="name" use="required" type="xs:string" />

   <xs:attribute name="value" use="required" type="xs:string" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="dynamicAttrs">

  <xs:annotation>

   <xs:documentation>Definition of the query dynamic attributes and

    their default values

   </xs:documentation>

  </xs:annotation>

  <xs:complexType>

   <xs:sequence>

    <xs:element name="predefinedDynamicAttr" minOccurs="0">

     <xs:complexType>

      <xs:sequence>

       <xs:element ref="fed:dynamicQueryInterval" minOccurs="0" />

       <xs:element ref="fed:dynamicGeoLocation" minOccurs="0" />

      </xs:sequence>

     </xs:complexType>

    </xs:element>

    <xs:element name="dynamicAttr" maxOccurs="unbounded"

     minOccurs="0">

     <xs:complexType>

      <xs:attribute name="name" type="xs:string" />

      <xs:attribute name="value" type="xs:string" />

     </xs:complexType>

    </xs:element>

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="dynamicQueryInterval">

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" name="fromDateTime" type="xs:dateTime" />

    <xs:element minOccurs="0" name="toDateTime" type="xs:dateTime" />

    <xs:element minOccurs="0" name="intervalNowToPast" type="xs:int" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="dynamicGeoLocation">

  <xs:complexType>

   <xs:sequence>

    <xs:element name="latitude" type="xs:string" />

    <xs:element name="longitude" type="xs:string" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

</xs:schema>



Descriptive IDs Schema

<?xml version="1.0" encoding="UTF-8"?>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"

 elementFormDefault="qualified" targetNamespace="urn:fiestaiot:experiment:descriptiveids:xsd:1"

 xmlns:edid="urn:fiestaiot:experiment:descriptiveids:xsd:1">

 

 <xs:element name="ExpDescriptiveIDs">

  <xs:complexType>

   <xs:sequence>

    <xs:element ref="edid:FemoDescriptiveID" maxOccurs="unbounded" />

   </xs:sequence>

  </xs:complexType>

 </xs:element>

 

 <xs:element name="description" type="xs:string" />

 <xs:element name="name" type="xs:string" />

 

 <xs:element name="FismoDescriptiveID">

  <xs:complexType>

   <xs:sequence>

    <xs:element minOccurs="0" ref="edid:description" />

    <xs:element minOccurs="0" ref="edid:name" />

   </xs:sequence>

   <xs:attribute name="id" type="xs:anyURI" />

  </xs:complexType>

 </xs:element>

 

 <xs:element name="FemoDescriptiveID">

  <xs:complexType>

   <xs:sequence>

    <xs:element maxOccurs="1" minOccurs="0" ref="edid:description" />

    <xs:element minOccurs="0" ref="edid:name" />

    <xs:element ref="edid:FismoDescriptiveID" />

   </xs:sequence>

   <xs:attribute name="id" type="xs:anyURI" />

  </xs:complexType>

 </xs:element>

</xs:schema>

 

 

2.3 API Usage

The different exposed services described in the ERM API above are exposed at the following URLs:

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/saveUserExperiments

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/deleteUserExperiments

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/saveUserExperiment

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/deleteUserExperiment

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/saveExperimentServiceModelObject

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/deleteExperimentServiceModelObject

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/getALLUserExperiments

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/getAllUserExperimentsDescreptions

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/getExperimentDescreption

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/getExperimentModelObject

http://[HOST]:[PORT]/experiment.erm/rest/experimentservices/getExperimentServiceModelObject

3 EEE APIs for Advance Experimenters

There are several EEE APIs that are available for advanced experimenters. These are public APIs. An interactive documentation of the EEE APIs can be found in the following link: https://platform.fiesta-iot.eu/EEEApidocs/

The EEE APIs are divided into 5 categories, namely, Scheduling, Polling, Subscription, Monitoring and Accouting APIs.

3.1 Scheduling API

These set of APIs help in scheduling of a FISMO on the EEE.

Scheduling APIs

3.2 Polling API

These set of APIs are used to execute the FISMO on top of the EEE.

Polling APIs

3.3 Subscription API

These set of APIs help in subscribe to an existing FISMO on the EEE.

Subscription APIs

3.4 Accounting API

These set of APIs help to understand the number of executions of a FISMO on the EEE.

Accounting APIs

3.5 Monitoring API

These set of APIs help to understand the executions of a FISMO on the EEE.

Monitoring APIs

4 IoT-Registry API for Advanced Experimenters

In the previous section, we have showed the most straightforward way to define an experiment or application. It goes without saying that novel or non-technical users will feel more comfortable with it. Nonetheless, there is another option for advanced experimenters, who will be able to interact directly with the iot-registry.

As a matter of fact, this Meta-Directory, heart of the Fiesta−IoT Platform, is in charge of managing, storing and serving all the resource descriptions and observations (i.e. data streams) coming from the underlying testbeds. In order to open it outwards, we have defined a REST-based API, called iot-registry, which provides the set of interfaces that enables the access to the information held by the FIESTA-IoT federation of testbeds.

During the implementation of the IoT-Registry we have tried to follow the CRUD (Create, Read, Update and Delete) approach as much as possible, but in order to provide a more user-friendly interface, some of the functionalities are not fulfilling this philosophy. The REST web-service implemented is accepting and returning RDF documents in various formats, like JSON-LD, N3, RDF/XML, CSV, plain text, n-quads, etc.

An interactive documentation of this IoT-Registry API can be found in the following link: https://platform.fiesta-iot.eu/iot-registry/docs/api.html

4.1 Resources

Resources-related IoT Service Endpoint

4.2 Observations

Observations-relater IoT-Registry

4.3 Endpoints

One of the advantages behind the usage of a semantic model as the one proposed by the FIESTA-IoT ontology is that everything is connected, thus shaping a huge but single graph. For this, we include, as part of the resource descriptions, an IoT-service endpoint that expose the last observations gathered by a particular “sensing device”.

Endpoints IoT Registry Endpoint

4.4 Queries

Whereas in the first four points we have covered various features to get data in a more guided way, with the use of SPARQL-based queries we offer the freedom to tailor the request as much as the experimenter wants.

  • Store. This works as a kind of catalogue through which we can offer a number of preconfigured queries to others. At the very same time, everyone is welcome to bring new SPARQL queries to the “store”, by just registering them (through a POST message). This way, non-skilled users could have illustrative examples of how to extract data from the meta-directory and find inspiration to create their own ones.
  • Execute. On the other hand, you can directly execute a query through these endpoints.



Queries IoT Registry Endpoint


Regarding this last part, we present int the following tables a number of well-known SPARQL queries that we encourage you to take a look before defining your own ones from scratch. They are the outcome of a compilation of most of the queries that all partners have been using in all our activities.

SPARQL 1 - FIesta-IoT Asset Summary

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

from=<starting_date>
to=
<ending_date>

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>

accept: text/tab-separated-values

SPARQL query

PREFIX ssn: <http://purl.oclc.org/NET/ssnx/ssn#>
PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT  (COUNT(DISTINCT ?dep) AS ?deployments)
    (COUNT(DISTINCT ?dev) AS ?devices)
    (COUNT(DISTINCT ?sens) AS ?sensors)
    (COUNT(DISTINCT ?obs) AS ?observations)
    (COUNT(DISTINCT ?n) AS ?nodes)
WHERE {
  {?dep a ssn:Deployment}
UNION
  {?dev a ssn:Device}
UNION
  {?sens rdf:type/rdfs:subClassOf ssn:SensingDevice}
UNION
  {?obs a ssn:Observation}
}

Response

?deployments  ?devices      ?sensors      ?observations
4      1068   3465   13538026  

 

 

SPARQL 2 - Number of sensing devices per type (and testbed)

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

NONE

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>
accept: application/json

SPARQL query

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX ssn: <http://purl.oclc.org/NET/ssnx/ssn#>
PREFIX iot-lite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>
SELECT   ?deployment (COUNT(DISTINCT ?sensor) AS ?sensors) ?qk ?type
WHERE {
       ?deployment a ssn:Deployment .
       {
              ?dev ssn:hasDeployment ?deployment .
              ?dev ssn:hasSubSystem ?sensor
       }
       UNION
       {
              ?sensor ssn:hasDeployment ?deployment
       }
       ?sensor iot-lite:hasQuantityKind ?qkr .
       ?qkr rdf:type ?qk .
       ?sensor a ?type
}
GROUP BY ?deployment ?qk ?type
ORDER BY ?deployment

Response

{
    "vars": [
        "deployment",
        "sensors",
        "qk",
        "type"
    ],
    "items": [
        {
            "deployment": "https://platform.fiesta-iot.eu/iot-registry/api/testbeds/a1yp9GcKEPw37Bx5rslgRI4QLSNCwEwBatCIOe_W0dHZCmzj2WmkExz3qoNuvWg1pueAXn1Li0JrNjvBiQwV3Q==",
            "qk": "http://purl.org/iot/vocab/m3-lite#ChemicalAgentAtmosphericConcentrationAirParticles",
            "type": "http://purl.org/iot/vocab/m3-lite#GaseousPollutantSensor",
            "sensors": "101^^http://www.w3.org/2001/XMLSchema#integer"
        },
        {
            "deployment": "https://platform.fiesta-iot.eu/iot-registry/api/testbeds/a1yp9GcKEPw37Bx5rslgRI4QLSNCwEwBatCIOe_W0dHZCmzj2WmkExz3qoNuvWg1pueAXn1Li0JrNjvBiQwV3Q==",
            "qk": "http://purl.org/iot/vocab/m3-lite#RelativeHumidity",
            "type": "http://purl.org/iot/vocab/m3-lite#HumiditySensor",
            "sensors": "20^^http://www.w3.org/2001/XMLSchema#integer"
        }, …
    ]
}

 

 

SPARQL 3 - Resource discovery (all nodes)

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

NONE

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>
accept: application/json

SPARQL query

PREFIX iot-lite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

PREFIX ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

PREFIX geo:  <http://www.w3.org/2003/01/geo/wgs84_pos#>

PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

 

SELECT ?sensor ?qk ?deployment ?type ?unit ?lat ?long ?endp

WHERE {

      ?deployment a ssn:Deployment .

      {

             ?dev ssn:hasDeployment ?deployment .

             ?dev ssn:onPlatform ?platform.

             ?platform geo:location ?point .

             ?point geo:lat ?lat .

             ?point geo:long ?long .

             ?dev ssn:hasSubSystem ?sensor

      }

      UNION

      {

             ?sensor ssn:hasDeployment ?deployment .

             ?sensor ssn:onPlatform ?platform.

             ?platform geo:location ?point .

      ?point geo:lat ?lat .

      ?point geo:long ?long .

      }

             ?sensor iot-lite:hasQuantityKind ?qkr .

             ?qkr rdf:type ?qk .

             ?sensor iot-lite:hasUnit ?unitr .

      ?unitr rdf:type ?unit .

            

             ?sensor a ?type

              OPTIONAL {

             ?sensor iot-lite:exposedBy ?serv .

             ?serv iot-lite:endpoint ?endp .

      }

}

Response

{

    "vars": [

        "sensor",

        "qk",

        "deployment",

        "type",

        "unit",

        "lat",

        "long",

        "endp"

    ],

    "items": [

        {

            "endp": "https://platform.fiesta-iot.eu/iot-registry/api/endpoints/ETV6cJzgIY6ADaxMe-V_ccxE2ObNgEqPdayROKWuEaWw4bfl5T2gBR-EgEqQUKa5lWCZwfVk4ShkyZS15-uae9aZ6Fy-KqiLJtBfJi3CZl3VXANSGgaLGdw0ztjw2hE_S6IvqBWTlV_JVOx4t16ISy21LovqdEUfGZYU0EWWL7b94SWfWnSjJqNUWi2qdaKb^^http://www.w3.org/2001/XMLSchema#anyURI",

            "type": "http://purl.org/iot/vocab/m3-lite#ElectricalSensor",

            "unit": "http://purl.org/iot/vocab/m3-lite#Percent",

            "qk": "http://purl.org/iot/vocab/m3-lite#BatteryLevel",

            "sensor": "https://platform.fiesta-iot.eu/iot-registry/api/resources/W_68NYKZCkpFspcj2U0yxSyfvgv6TK4jRemmB9ajOw3sAWsJJz4rJnm-F-CH4eEkRTjN233Bx8w5fZliZw7rGE1tCxVLLLRaGt_0SbJhijysWfmUFhEAXGnawnyUslbRR4aiYGpOV3q7yW5_kqgYYA==",

            "long": "-3.80649^^http://www.w3.org/2001/XMLSchema#double",

            "lat": "43.46104^^http://www.w3.org/2001/XMLSchema#double",

            "deployment": "https://platform.fiesta-iot.eu/iot-registry/api/testbeds/a1yp9GcKEPw37Bx5rslgRI4QLSNCwEwBatCIOe_W0dHZCmzj2WmkExz3qoNuvWg1pueAXn1Li0JrNjvBiQwV3Q=="

        }, ...

      ]

   

 

 

SPARQL 4 - Resource discovery (phenomena + location filters)

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

NONE

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>

accept: text/tab-separated-values

SPARQL query

PREFIX iot-lite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

PREFIX ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

PREFIX geo:  <http://www.w3.org/2003/01/geo/wgs84_pos#>

PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

 

SELECT ?dev ?sensor ?qk ?unit ?endp ?lat ?long

WHERE {

    ?dev a ssn:Device .

 

    ?dev ssn:onPlatform ?platform .

    ?platform geo:location ?point .

    ?point geo:lat ?lat .

    ?point geo:long ?long .

    ?dev ssn:hasSubSystem ?sensor .

    ?sensor iot-lite:hasQuantityKind ?qkr .

    ?qkr rdf:type ?qk .

    ?sensor iot-lite:hasUnit ?unitr .

    ?unitr rdf:type ?unit .

    OPTIONAL {

        ?sensor iot-lite:exposedBy ?serv .

        ?serv iot-lite:endpoint ?endp .

    }

    VALUES ?qk {m3-lite:AirTemperature

        m3-lite:TemperatureSoil

        m3-lite:Illuminance

        m3-lite:AtmosphericPressure

        m3-lite:RelativeHumidity

        m3-lite:WindSpeed

        m3-lite:SoundPressureLevel

        m3-lite:SoundPressureLevelAmbient

        m3-lite:Sound

        m3-lite:SolarRadiation

        m3-lite:ChemicalAgentAtmosphericConcentrationCO

        m3-lite:chemicalAgentAtmosphericConcentrationO3

        }.

} order by asc(UCASE(str(?qk))) LIMIT 10

Response

{

    "vars": [

        "dev",

        "sensor",

        "qk",

        "unit",

        "endp",

        "lat",

        "long"

    ],

    "items": [

        {

            "qk": "http://purl.org/iot/vocab/m3-lite#AirTemperature",

            "endp": "https://platform.fiesta-iot.eu/iot-registry/api/endpoints/hwY7kOlDdwluzg7c7VfQgJ5s6hwpkqPXIzkHMlJYgq6drrQlLqor9vFkRVkOl6IHkOFbPnW2XhJDMnwFXWkdsuzYnC0eI1NbSu1HaCk1SdWSfsfsdW-xBkDoxLautQqvOMjBEF0R44FPapaawuS98cmKOIsFEral7nfcuNOaKE-26_fBxQ8fZVzZMZok0dW2yhuPpUwK1BzzZe1eXVdVVA==^^http://www.w3.org/2001/XMLSchema#anyURI",

            "unit": "http://purl.org/iot/vocab/m3-lite#DegreeCelsius",

            "sensor": "https://platform.fiesta-iot.eu/iot-registry/api/resources/9Ww6ZciSwY6B-Iv_gQafrL1cYcvkif3P4ZfTaYP0kC5Xh7PlicakkcSefMICw5Godss8sBuDPYsBWxwYaE8-vT_7Asmd3F9ai1Y9jQpvMeuiFDaLQJTuw7MNeyZ1Q4fB87HpKAu_vEhUNlJkA4vKEQ==",

            "long": "-3.80429^^http://www.w3.org/2001/XMLSchema#double",

            "lat": "43.46346^^http://www.w3.org/2001/XMLSchema#double",

            "dev": "https://platform.fiesta-iot.eu/iot-registry/api/resources/-58RQD27b6aKsU46_YG7yg3Dt7KZHIAHI57K6OPA0_kqOexdipyqLEjLO3r3Hd7XxCR2B4qGftmCx4xucduHILJhq5jFqVqKpYm_hlnlLyv6jKzMmbmKojx6R4_X1Ep9"

        }, …

    ]

}     

 

 

SPARQL 5 - Last observation (phenomena+location filters)

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

from=<starting_date>
to=<ending_date>

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>

accept: application/json

SPARQL query

Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

Prefix time: <http://www.w3.org/2006/time#>

PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

select ?sensorID (max(?ti) as ?time) ?value ?latitude ?longitude ?qk

where {

    ?o a ssn:Observation.

    ?o ssn:observedBy ?sensorID.  

    ?o ssn:observedProperty ?qkr.

    ?qkr rdf:type ?qk.

    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient m3-lite:AirTemperature}

    ?o ssn:observationSamplingTime ?t.

    ?o geo:location ?point.

    ?point geo:lat ?latitude.

    ?point geo:long ?longitude.

    ?t time:inXSDDateTime ?ti.

    ?o ssn:observationResult ?or.

    ?or  ssn:hasValue ?v.

    ?v dul:hasDataValue ?value.

    {

        select  (max(?dt)as ?ti) ?sensorID        

        where {

            ?o a ssn:Observation.

            ?o ssn:observedBy ?sensorID.  

            ?o ssn:observedProperty ?qkr.

            ?qkr rdf:type ?qk .

            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient m3-lite:AirTemperature}

            ?o ssn:observationSamplingTime ?t.

            ?t time:inXSDDateTime ?dt.

        }group by (?sensorID)

    }

    FILTER (

       (xsd:double(?latitude) >= "-90"^^xsd:double)

    && (xsd:double(?latitude) <= "90"^^xsd:double)

    && ( xsd:double(?longitude) >= "-180"^^xsd:double) 

    && ( xsd:double(?longitude) <= "180"^^xsd:double)

    ) 

} group by ?sensorID ?time ?value ?latitude ?longitude ?qk

Response

{

    "vars": [

        "sensorID",

        "time",

        "value",

        "latitude",

        "longitude",

        "qk"

    ],

    "items": [

        {

            "longitude": "-5.868359516962779E-1^^http://www.w3.org/2001/XMLSchema#double",

            "qk": "http://purl.org/iot/vocab/m3-lite#AirTemperature",

            "value": "17.1578786035^^http://www.w3.org/2001/XMLSchema#float",

            "sensorID": "https://platform.fiesta-iot.eu/iot-registry/api/resources/CEjjXic0G1A-N4qCyRy-jxBL4H3HorsFQ49UB9MaKK54vkN15o_Ycq4oNIxBwfnyiMgQC6sTk73J21rsIal2CFSiqSEq5HpRZDAk8oJuUpLNMfGOtOdaWluFBsvAaTWgQADCRbPGBJ-MvTe9NIhnEQ==",

            "latitude": "5.124193338767561E1^^http://www.w3.org/2001/XMLSchema#double",

            "time": "2017-09-13T15:21:39.766370+00:00^^http://www.w3.org/2001/XMLSchema#dateTime"

        },...
     ]
}   
 

 

 

SPARQL 6 - Last observation (from a sensor ID)

Request

GET 1

URL Parameters

NONE

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>

accept: text/tab-separated-values

SPARQL query

Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

Prefix iotlite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

Prefix time: <http://www.w3.org/2006/time#>

PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

select ?sensorID (max(?ti) as ?time) ?value ?latitude ?longitude ?qk

where {

    ?o a ssn:Observation.

    ?o ssn:observedBy ?sensorID.  

    ?o ssn:observedProperty ?qkr.

    ?qkr rdf:type ?qk.

    Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient m3-lite:AirTemperature}

    VALUES ?sensorID {<https://platform.fiesta-iot.eu/iot-registry/api/resources/0wM31oVlhH7JC8Kg0BRcFXxpDXwefGiJbx8FYgIaBx5x0JQMhS7W2ra8B7b6tAqnl2tvHV7CZh1t1DnMfTmn9Q==>    }

    ?o ssn:observationSamplingTime ?t.

    ?o geo:location ?point.

    ?point geo:lat ?latitude.

    ?point geo:long ?longitude.

    ?t time:inXSDDateTime ?ti.

    ?o ssn:observationResult ?or.

    ?or  ssn:hasValue ?v.

    ?v dul:hasDataValue ?value.

    {

        select  (max(?dt)as ?ti) ?sensorID        

        where {

            ?o a ssn:Observation.

            ?o ssn:observedBy ?sensorID.  

            ?o ssn:observedProperty ?qkr.

            ?qkr rdf:type ?qk .

            Values ?qk {m3-lite:Sound m3-lite:SoundPressureLevelAmbient m3-lite:AirTemperature}

            ?o ssn:observationSamplingTime ?t.

            ?t time:inXSDDateTime ?dt.

        }group by (?sensorID)

    }  

} group by ?sensorID ?time ?value ?latitude ?longitude ?qk

Response

{

    "vars": [

        "sensorID",

        "time",

        "value",

        "latitude",

        "longitude",

        "qk"

    ],

    "items": [

        {

            "longitude": "-3.12E0^^http://www.w3.org/2001/XMLSchema#double",

            "qk": "http://purl.org/iot/vocab/m3-lite#AirTemperature",

            "value": "19.5295470384^^http://www.w3.org/2001/XMLSchema#float",

            "sensorID": "https://platform.fiesta-iot.eu/iot-registry/api/resources/0wM31oVlhH7JC8Kg0BRcFXxpDXwefGiJbx8FYgIaBx5x0JQMhS7W2ra8B7b6tAqnl2tvHV7CZh1t1DnMfTmn9Q==",

            "latitude": "3.989E1^^http://www.w3.org/2001/XMLSchema#double",

            "time": "2017-09-13T15:19:28.919959+00:00^^http://www.w3.org/2001/XMLSchema#dateTime"

        }

    ]

   

 

 

SPARQL 7 - Observations within a time window

Request

GET https://platform.fiesta-iot.eu/iot-registry/api/queries/execute/global

URL Parameters

NONE

HTTP Headers:

Content-Type: text/plain
iPlanetDirectoryPro: <ssoToken>

accept: text/tab-separated-values

SPARQL query

Prefix ssn: <http://purl.oclc.org/NET/ssnx/ssn#>

Prefix iot-lite: <http://purl.oclc.org/NET/UNIS/fiware/iot-lite#>

Prefix dul: <http://www.loa.istc.cnr.it/ontologies/DUL.owl#>

Prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

Prefix time: <http://www.w3.org/2006/time#>

Prefix m3-lite: <http://purl.org/iot/vocab/m3-lite#>

Prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

select ?s ?ti ?value ?lat ?lon ?qk ?u

where {

    ?o a ssn:Observation.

    ?o ssn:observedBy ?s.

    ?s iot-lite:hasUnit ?unit.

    ?unit rdf:type ?u .   

    ?o geo:location ?point.

    ?o ssn:observedProperty ?qkr.

    ?qkr rdf:type ?qk.

    ?point geo:lat ?lat.

    ?point geo:long ?lon.    

    ?o ssn:observationResult ?or.

    ?or  ssn:hasValue ?v.    

    ?v dul:hasDataValue ?value.    

    ?o ssn:observationSamplingTime ?t.

    ?t time:inXSDDateTime ?ti.

    FILTER((xsd:dateTime(?ti) > "2017-09-12T00:00:00.000Z"^^xsd:dateTime) && (xsd:dateTime(?ti) < "2017-09-13T00:00:00.000Z"^^xsd:dateTime))

   

} group by ?s ?ti ?value ?lat ?lon ?qk ?u

order by ?s desc(?ti)

Response

{

    "vars": [

        "s",

        "ti",

        "value",

        "lat",

        "lon",

        "qk",

        "u"

    ],

    "items": [

        {

            "lat": "5.1242957667395444E1^^http://www.w3.org/2001/XMLSchema#double",

            "lon": "-5.88064433794546E-1^^http://www.w3.org/2001/XMLSchema#double",

            "qk": "http://purl.org/iot/vocab/m3-lite#SolarRadiation",

            "ti": "2017-09-13T15:24:52.900773+00:00^^http://www.w3.org/2001/XMLSchema#dateTime",

            "value": "1848.88523785^^http://www.w3.org/2001/XMLSchema#float",

            "u": "http://purl.org/iot/vocab/m3-lite#WattPerSquareMetre",

            "s": "https://playground.fiesta-iot.eu/iot-registry/api/resources/-6bRByQeMiw8AnpjWf-htvUZwCmgI_Bm_QVba0aXULczKX0LNZR6i0nNFQy04x8-2eNU7yHehLih2qTOYFcxyA3OrlG013eJrAqJ7pwa0plxAETgyc60Xe63ZzJiUZmcAFZnU68qcAeR8oT5M8F7ZA=="

        }, ...

    ]
}
     

Aside from this handbook, we have also published them in: http://moodle.fiesta-iot.eu/mod/page/view.php?id=146\cf0\par