Overview

"Backend" Algorithms

"Back-end" components are responsible for the retrieval of data from the physical devices. These components also manipulate the data in such a way that irrespective of the format and specifics of the "raw" data read from the devices, the data stored in the "readout" tables in the main site database is consistent. For example, an actual device may report energy in kWh, however these values are manipulated by the back-end so that the numbers in the "readout" tables represent Joules. The actual timestamp of the data may also be manipulated by the backend so that the data in the readout tables always comes with a timestamp that falls on regular intervals. So a data that actually refers to a reading taken at 14:00:24.0212 is actually stored in the readout tables with a timestamp of 14:00:00. This makes it easier for subsequent calculations to run, and for the data to be graphed etc. 

"Site" Algorithms

The data loaded by the backend into the readout tables is then "aggregated" by algorithms (stored routines) in the site MySQL databases. We've internally referred to the process of data-processing done mainly within the site MySQL databases as "data aggregation." A more appropriate term would probably have been "data pre-processing" since aggregation is only a part of the transformation done by the site to ensure that the readout data is in a suitable format for consumption by the website. 

Essentially, the following calculations/transformations are done:

• Device-specific data is "aggregated" up to hourly, daily, monthly values
• Device-specific data is combined to form "system-level" readings, so for example, a given system may have a reading at 14:00 which includes the data from devices measuring momentary power, cumulative energy, temperature, irradiance, etc. 
• "System-level" readings for subsystems (systems with parent systems) are aggregated together to obtain "aggregator-system" readings. For example, if a site has 3 subsystems, then the energy readings for a given moment are added together to obtain the energy reading for the whole site at that moment in time.
• Certain values are calculated from the different types of readings, such as: estimated energy, effective availability, weather-corrected predicted power.