# dbus_vrm

This repository contains two services that run on the
[Color control](http://www.victronenergy.com/panel-systems-remote-monitoring/colorcontrol/).

- **vrmlogger**: The vrm-logger sends data from the D-Bus on the ccgx to the vrm database, or stores it on a
usb/sdcard in a format that can be manually uploaded to the VRM Portal. Sending is done with HTTP(S) POST
request.
- **vrmpubnub**: Pubsub client that uses the commercial pubsub provider Pubnub. Used by Remote VEConfigure
and also to send live data updates to apps and website.

The datamap between dbus and VRM Database is defined in [dataAttributes en deviceTypes.xlsm](https://svn.victronenergy.com/viewvc/vrm_database/trunk/dataAttributes%20en%20deviceTypes.xlsm?revision=511&view=co).
That excelsheet has a macro to generate the code in datalist.py

The vrm-log-backend API documentation is [here](https://svn.victronenergy.com/viewvc/vrm_database/trunk/documentation/VRM%20Database%20-%20API%20for%20gateways.docx?view=markup)

## vrmlogger architecture

The application takes data from the D-Bus, and sends it to either one of the two possible endpoints:
- HttpEndpoint
- StorageEndpoint

The HttpEndpoint takes care of sending the data to the VRM Portal (to the vrm-log-backend to be exact). It
uses one worker thread to take care of sending the data, and a Queue to pass the data to that thread. See
http_endpoint.py for details.

The main program, vrmlogger, initialises a NonVolatileQueue, called 'buffer', with a maximum age of two days.
This is the backlog buffer, stored on disk in an sqlite3 database. It is persistent during changes from
LOGMODE.LogToStorage to LOGMODE.LogToVrm.

### D-Bus to data-attribute-code Datamap
In the datamap, there is a list of D-Bus service names (com.victronenergy.vebus, com.victronenergy.battery,
etc) that contain values that need to be sent to the VRM Database. These are called the deviceTypes. Then
there is a long list, containing all the mappings. Each line in the mapping contains:

  * Link to dbus server name: com.victronenergy.vebus
  * Path to busitem: /Dc/0/V
  * Code to use when communicating this value to VRM Database: 'V'
  * Type of data:
      - Config change. Used for firmware versions, product ids, etc. All config values are sent on powerup,
      and afterwards on change. When one changes, resend all of them. (Excel column Whentolog == configChange)
      - Normal value. Voltages, currents, etcetera. Whentolog = onIntervalAlwaysAndOnEvent or onIntervalAlways
      or onIntervalOnlyWhenChanged)
      - Hourly delta. Used for kWh-deltas. Sent on the passing of the hour, and data is added to each dataSend
      of the normal values (Excel column targetTable == logDataHourlyDeltas)

The normal values are split up in three categories, which define when this data needs to be sent:

  - Some parameters are logged everytime the interval passes. Interval default is 15 minutes. Examples are
  battery voltages, currents, etcetera. (onIntervalAlways)
  - Other parameters, for example H1 to H16 from the BMV, are included in the log only when they are
  different from last time. (onIntervalOnlyWhenChanged)
  - A change of some parameters will trigger an immediate log. For example when an inverter switches from
  inverting to charging, you want to see that immediately. (onIntervalAlwaysAndOnEvent)

When normal values are sent, all of them are sent together. No partial sends. This is to allow calculations
to made server-side in the vrm-log-backend. On a configChange, also all parameters with type configChange are
resent. Exceptions to this are values marked with onIntervalOnlyWhenChanged.

### Data-attribute details
Following generic codes are mandatory for each request sent:

  * 'd': gatewayid. For CCGX set this to 1.
  * 'IMEI': VRM Portal identifier. For CCGX set this to lower case mac-address stripped of the colons, for
  example 0013d3dbc23b
  * 'c': type of command. 0=announce (boot up message), 1=log (data report), 2=config change
  * 't': logging interval in seconds

And these is one optional code for the data reports (c=1):

  * 'TO': time-offset in seconds. Only add this code when sending data from the backlog. Formula to use:
  now() - unix_time_stamp_taken_on_clone_of_data. Server side the unix_time stamp insterted into the
  database is calculated as now() - TO.

Instancing: multiple instances of the same device, for example a battery monitor, can be available on the
D-Bus. Each has a unique instance (/DeviceInstance/). This is added to the code like this:

    v[0]=12.2&v[4]=24.3

### Interval and timing notes

  * If a value that is configured with onIntervalAlwaysAndOnEvent or configChange changes, don't
  immediately initiate sending all data, since often a lot of values will change at the same time. Start a
  short timer instead, and then send the data.
  * Above type of sending data (on an event) should also reset the interval-timer to 0.
  * Logs are always to be sent in chronological order.



