Domoticz HTTP/HTTPS poller and JSON

Adding a sensor of a remote host to Domoticz is easy, just not very well documented.

I have a DS1820 one-wire sensor plugged on a Linux machine that polls the temperature every minute then stores it in a CSV. Since I recently installed Domoticz on a Raspberry PI, I wanted to show the values there as well. I’ll show what you need to do this.

Open the Setup menu and choose the Hardware option. Enter a name you like and select the HTTP/HTTPS poller type, you should see the following:

HTTP/HTTPS poller to handle JSON in Domoticz
HTTP/HTTPS poller to handle JSON in Domoticz

Data Timeout: leave it Disabled

ContentType: the type of received data, application/json

URL: the full URL including scheme, host name and path where you will have the JSON file with the latest data.

Command: this is a filename (with extension) for a script that will handle the downloaded JSON, which Domoticz will search at …/domoticz/scripts/lua_parsers/<command>
Tehre’s a basic example called example_json.lua, for a this setup it will be fine.

Refresh: refresh interval in seconds.

After adding you should see it immediately in the list:

HTTP/HTTPS poller added in Domoticz
HTTP/HTTPS poller added in Domoticz

Click on Create Virtual Sensors button to add a sensor, give it a custom name, select the type of data you want to log and select OK. It should appear in your devices list:

A temperature sensor using Domoticz's HTTP/HTTPS poller
A temperature sensor using Domoticz’s HTTP/HTTPS poller

Note the Idx of the sensor (not the ID nor Unit, the first number) which is 15 in my case. You will need to put this in the JSON file on remoteserver in a field named id – yes, this is inconsistent.

The JSON file you need to provide on remoteserver can be as simple as this:

{ "id": 15, "temperature": 12.3 }

Tip: you can output this with any script, you can consider it a string, just replace the value with the latest measurement. You also need to update it periodically.

As soon as you added the poller, Domoticz starts to poll the URL you supplied and passes the result to the script you supplied.

If in the next few minutes you start to see the data in Domoticz, congratulations, you have it working. If not, check the Troubleshooting section below.

Note 1: Domoticz executes the script one time for every HTTP/HTTPS poller “hardware” and you can have multiple sensors configured for one poller. In this case you need to supply multiple id/idx values and multiple values for measurements and need a custom script updating each sensor.

Note 2: In the example script the id is read from the JSON and blindly supplied to Domoticz which will update the sensor having that as idx. This may overwrite the data of any other sensor if configured improperly.

Unfortunately Domoticz is only supplying the hardware idx as hwdId variable but not the idx of the sensor(s) on that poller, so if you want to prevent it you need to hardcode the idx of the sensor(s) in your script, e.g. you can create remote_1_json.lua with the content:

local idx = 15
local temperature = domoticz_applyJsonPath(request['content'], '.temperature')
domoticz_updateDevice(idx, '' , temperature)

Then you can entirely skip the id field in JSON:

{ "temperature": 12.3 }

Just make sure you update the script’s name on the web interface.

Note 3: you might need to add the following header on the remote host to your response:

Content-Type: application/json

Troubleshooting

Error: CLuaHandler (updateDevice from LUA) : Incorrect parameters type 

One of the field received in JSON is invalid. The id should be an integer, temperature should be a float.

Error: CLuaHandler: cannot open /home/.../domoticz/scripts/lua_parsers/example_json: No such file or directory 

You need to provide a full file name with extension in the Command field of the Setup > Hardware page. Also, make sure the script exists.

CLuaHandler (updateDevice from LUA) : idx=15 nvalue= svalue=12.3 invalue=0 signallevel=12 batterylevel=255 

This is normal, the script parsed the JSON and issued an update for the sensor. If you can’t see the update on the interface, double check the idx or id parameter you have in the JSON file.

GameBoy hardware hacking: Part 1

In the case of GameBoy and GameBoy Color systems the console itself holds just a really minimal program on the board, the so-called boot ROM (256 bytes long). It does the initialization of the components and the cartridge, also it reads the header part of the program stored on the cartridge and checks if that is valid. (Fun fact: the scrolling Nintendo logo is read from the cartridge and it is really a test to see if the cartridge can be accessed correctly. The logo is then compared to the one stored in the boot ROM to see if it was read correctly, if not that indicates some pins are not working and the CPU halts. This is when you need to blow the cartridge.).

All the other program code (i.e. the game itself and all the instructions to make it work) is located on the GameBoy Cartridge alongside with the textures, sounds and all the rest.

So the cartridge is basically the part of the system, the CPU reads it directly when it needs the next instruction and executes the code from there without copying it to its own memory.

The cartridges usually consist of the following components:

  • ROM: Read Only Memory
    Stores the program code and all the resources needed for the game. It is programmed once and then never written again.
  • MBC: Memory Bank Controller (optional)
    The Memory Bank Controller switches banks of the addressable memory space (32 kilobyte in total). The lower 16 kilobytes are always the first 16 kilobytes of the ROM while the upper 16 kilobytes are switchable between several blocks (i.e. part of the ROM or the RAM). There are several MBCs out there with their own purposes. This is a really long story, I’ll write more about this later.
  • RAM: Random Access Memory (optional)
    This is a readable and writable storage, when a game has high-score list, saved game state, etc. it holds them information in the RAM. This is a standard SRAM so this is volatile – it needs constant power to keep the data alive this is why cartridges with RAM need batteries as well.
  • Battery: (optional)
    The battery provides power for the RAM to keep the stored data alive.
  • Connector:
    Last but not least, all the cartridges have a 32 pin connector that provides a connection between the console and the components of the cartridge.

The interface between the console and the cartridge consists of 32 pins (direction is written from console perspective, out: from console to cartridge, in: from cartridge to console):

  • pin 1 [out]: VCC (+5 V)
    This is the supply pin for the cartridge and its components (i.e. the ROM, RAM, MBC).
  • pin 2 [n/c]: not connected or CPU clock
    I have not seen this pin connected so far but some documents claims this is a direct output of the CPU clock. I ran a logic analyzer on this pin, it showed a high-low pattern alternating at 1.04 MHz (0.46 µs high followed by 0.50 µs low).
  • pin 3 [out]: !write
    This is a signal line that is low when a write request is made to RAM. Otherwise this is high.
  • pin 4 [out]: !read
    This is a signal line that is low when a read reqeust is made to ROM, MBC or RAM. Otherwise this is high.
  • pin 5 [out]: !RAM select
    This line is low when the RAM or the MBC is being accessed. Both of them supports reads and writes. Otherwise this is high.
  • pin 6..21 [out]: address selection bit 0..15
    These bits sets the address that is being written or read in the RAM, ROM, or in the MBC registers.
  • pin 22-29 [out/in]: data in/out bit 0..7
    When the console wants to read from the ROM, RAM or MBC these pins are inputs, their values being set by the remote component. When the console wants to write data these are outputs, set to the data by the console.
  • pin 30 [out]: !reset
    When the console reboots it sets this pin high after about 4-5 ms. This delay is probably introduced to give the clock a few cycles to stabilize.
  • pin 31 [n/c]: not connected or analog audio input
    I have not seen this pin connected so far but some documents claims this is an analog audio input. (I think I’ve read something about a direct sound channel on the GameBoy CPU, but that might be a different story.)
  • pin 32 [out]: GND
    The ground connection for the cartridge and all the components.

In the following posts I will write about the different cartridge and MBC types, and also will post schematics and circuits to build a few different homebrew GameBoy cartridges. Stay tuned.

Multiplayer gaming, part 1: Sync the future!

Last year I was writing a little game intended to be run in the browser. I was writing it in pure javascript (i.e. no graphics or whatsoever was stored, everything was dynamically generated) for the js13k Games 2013, it is the Untitled 13 (can be played here).

One of the base concepts was to build a multiplayer game. It was challenging despite I’ve written some basic games, also mulitiplayer ones, I’ve never tried to write a real-time strategy game yet (not counting a quick POC). The synchronization of the game between players was the most interesting part of it, and therefore my favorite one. I was not intended to sync the whole game state (map, objects, internal states, etc.) continuously as this can easily become a huge set of data even for tiny games at high refresh rates so I decided I will come up with a better, more resource-friendly solution.

I’ve been thinking about the game: it has some mechanics, player events events and also their effects on the gameplay and concluded one could be separated from the other – of course it turned out that I was not the first who came up with this approach. So instead of syncing everything, I was thinking about synchronizing just the map in its starting state and a list of events.

All these events were always originated from the players (i.e. buy an item, toggle a switch) or from the game mechanics (i.e. an enemy is close enough to be fired on, an object loses health, objects collide). As the game on both ends are always identical the result of mechanics should be always identical too, therefore all I left to deal with were the events from the players.

So basically all I was doing is this: my game had so-called “ticks” that were the updates for the game internal states (this event was happening with a guaranteed interval, i.e. 15 times per seconds, or if late (due to lack of CPU resources or bad timing) all the affected ticks ran anyways to catch up), and when a player did an action (i.e. clicked a switch) I stored a record in a queue about it and a “process this on the nth tick” property. I’ve also sent this record over the network to the other player. This way I introduced a slight delay (this nth tick was always about 300 milliseconds in the future) but during this delay I sent the event to the other player and when the nth tick came both players made the change on their isolated environment in the exact same moment (speaking in game time) and therefore both ends have seen the same new state, the game is deterministic.

Later I’ve learned – in a Gaffer on Games post and in another one about network programming in Age of Empires (both are must reads!) – this is a so called simultaneous simulation approach with one big difference. The handling of the lag on the network. This was the worst thing during the development and tests of the game. As I’ve been syncing future events if the notification got stuck somewhere between the players it could easily come later than it must have been executed. As the tick it was intended to be run at was passed (and the other player have executed at that time) the game became out of sync. And this is bad, really bad. (I’ve made some efforts to handle this, but these turned the game code much more complicated, i.e. to be sure that important events happen on both sides I added them as an event to the queue (and sent that over the network), that’s why the little creatures pause for a while before exploding at the target markers. But this is bad, don’t do this. Keep reading for a simpler solution instead.)

In a real simultaneous simulation the players should be aware of the lag in the propagation of events so instead of just saying “here, run this on the nth tick” there needs to be a “I have nothing for you to run on the nth tick, carry on”. And if the other player does not specify what to do (or not to do) on the nth tick when it comes the simulation should pause while it is unsure about what to do. Which is not nice but far much better than becoming out of sync. Obviously working with 15 ticks per seconds and expecting the network to transfer the data for the next tick is nearly impossible (talking about transmitting over the internet as this would permit about a 66 milliseconds deadline on a client1-server-client2 route, also sending at least 15 events every second) and would result frequent pauses, and that would be really annoying. So instead of scheduling events for any tick of the game a rule should be applied to limit the ticks that can accept events. Saying events can be scheduled only for every 5th ticks reduces the count of transmitted packets dramatically, also extends the deadline to a more sane value.

Also, when a pause occurs due to network lag the rule of scheduling could change i.e. not every 5th but every 10th tick should accept events. And likewise when the network seems reliable for a long time this number could decrease.

But this topic will be covered in another post, first I’ve to try this approach. Although I think this should work nicely as games like Age of Empires and Warcraft used this method pretty reliably.

Recommended reading:

Game Boy: my first steps

On 4th March, 2013 I bought a Game Boy Color in really good condition, so this became my very first game console (at age 27, huh).

On this day I heard the welcome notes for the first time – it felt great, hehe.

It says “© 1998 Nintendo” on the back, so this is a 15 years old platform (actually mostly a 24 years old with some improvements, as the original Game Boy was released back in 1989), but still great! I’m in love with it.

I’ve also checked out the Game Boy Advance (GBA) before I bought decided to go with GB/GBC line – the GBA has a 32-bit CPU at 16 MHz and 288 kB RAM + 96 kB of VRAM, this spec is just ridiculously high, who needs this much anyways? I’m sticking with the lower specs.

So I am now very happy with this little Game Boy Color – next thing on my TODO list to buy a programmable cartridge and start to test my developments outside the emulator.

This post was migrated and edited a bit from my old blog where it was posted on 1st April, 2013.

Development for Game Boy (Color)

I have to say, the Game Boy is a pretty impressive platform!

With a 4 MHz 8-bit CPU, 8 kB RAM and 8 kB video RAM it has strict limitations but some nice perspectives at the same time.

It was a rather unexpected fact to me that the 160×144 pixels screen is refreshed at 60 FPS (!) and the device has a 4 channel sound system with mono speaker (but stereo headphone jack). It has also hardware-level tile storing (8×8 or 8×16 tile data, the pixel data) and tile mapping (32×32 tiles map, that’s 256×256 pixels on this bad boy:) ) function. Also, the background-scrolling is implemented in hardware, at any given time 20×18 tiles (160×144 pixels) are visible but 32×32 tiles (256×256 pixels) are in the video RAM.

Documentations I find useful and any work I do will be stored in a git repository on GitHub: https://github.com/gheja/gameboydev/

This post was migrated and edited a bit from my old blog where it was posted on 30th March, 2013.