Domoticz HTTP/HTTPS poller and JSON

Adding a sensor of a remote host to Domoticz is easy, just not very well documented.

I have a DS1820 one-wire sensor plugged on a Linux machine that polls the temperature every minute then stores it in a CSV. Since I recently installed Domoticz on a Raspberry PI, I wanted to show the values there as well. I’ll show what you need to do this.

Open the Setup menu and choose the Hardware option. Enter a name you like and select the HTTP/HTTPS poller type, you should see the following:

HTTP/HTTPS poller to handle JSON in Domoticz
HTTP/HTTPS poller to handle JSON in Domoticz

Data Timeout: leave it Disabled

ContentType: the type of received data, application/json

URL: the full URL including scheme, host name and path where you will have the JSON file with the latest data.

Command: this is a filename (with extension) for a script that will handle the downloaded JSON, which Domoticz will search at …/domoticz/scripts/lua_parsers/<command>
Tehre’s a basic example called example_json.lua, for a this setup it will be fine.

Refresh: refresh interval in seconds.

After adding you should see it immediately in the list:

HTTP/HTTPS poller added in Domoticz
HTTP/HTTPS poller added in Domoticz

Click on Create Virtual Sensors button to add a sensor, give it a custom name, select the type of data you want to log and select OK. It should appear in your devices list:

A temperature sensor using Domoticz's HTTP/HTTPS poller
A temperature sensor using Domoticz’s HTTP/HTTPS poller

Note the Idx of the sensor (not the ID nor Unit, the first number) which is 15 in my case. You will need to put this in the JSON file on remoteserver in a field named id – yes, this is inconsistent.

The JSON file you need to provide on remoteserver can be as simple as this:

{ "id": 15, "temperature": 12.3 }

Tip: you can output this with any script, you can consider it a string, just replace the value with the latest measurement. You also need to update it periodically.

As soon as you added the poller, Domoticz starts to poll the URL you supplied and passes the result to the script you supplied.

If in the next few minutes you start to see the data in Domoticz, congratulations, you have it working. If not, check the Troubleshooting section below.

Note 1: Domoticz executes the script one time for every HTTP/HTTPS poller “hardware” and you can have multiple sensors configured for one poller. In this case you need to supply multiple id/idx values and multiple values for measurements and need a custom script updating each sensor.

Note 2: In the example script the id is read from the JSON and blindly supplied to Domoticz which will update the sensor having that as idx. This may overwrite the data of any other sensor if configured improperly.

Unfortunately Domoticz is only supplying the hardware idx as hwdId variable but not the idx of the sensor(s) on that poller, so if you want to prevent it you need to hardcode the idx of the sensor(s) in your script, e.g. you can create remote_1_json.lua with the content:

local idx = 15
local temperature = domoticz_applyJsonPath(request['content'], '.temperature')
domoticz_updateDevice(idx, '' , temperature)

Then you can entirely skip the id field in JSON:

{ "temperature": 12.3 }

Just make sure you update the script’s name on the web interface.

Note 3: you might need to add the following header on the remote host to your response:

Content-Type: application/json

Troubleshooting

Error: CLuaHandler (updateDevice from LUA) : Incorrect parameters type 

One of the field received in JSON is invalid. The id should be an integer, temperature should be a float.

Error: CLuaHandler: cannot open /home/.../domoticz/scripts/lua_parsers/example_json: No such file or directory 

You need to provide a full file name with extension in the Command field of the Setup > Hardware page. Also, make sure the script exists.

CLuaHandler (updateDevice from LUA) : idx=15 nvalue= svalue=12.3 invalue=0 signallevel=12 batterylevel=255 

This is normal, the script parsed the JSON and issued an update for the sensor. If you can’t see the update on the interface, double check the idx or id parameter you have in the JSON file.

Bokosan – a reverse Sokoban

Bokosan is a reverse Sokoban where boxes are on a goal position and the player needs to pull them away while avoiding deadly spikes and falling into pits.

The game has 20 levels, single player mode, time challenge mode with online leaderboard, customizable player appearance, sound, 4 music tracks (6 in full version) and a few more features.

It is an entry for the js13kgames gamedev competition, where the goal is to create a web browser game using only JavaScript/CSS/HTML, with final size (after ZIP compression) below 13 kB, in a month. It was created for the 2015 jam for the theme “reverse”.

This year I worked together with my brother, Zsolt, he composed the music and created most of the levels.

Play it here.

Graphics

I’ve decided to go with raster graphics, stored in PNG with low color count and replacing colors in code if needed. I’ve tried several other methods to store graphics (putting each sprite in a string, etc.) but the overhead for decoding was just too big.

(The image is scaled down, click here for full version.)

The final PNG contains the Jostix font by Ray Larabie (only uppercase letters, numbers, punctuation marks), 13 tiles, 9 character animation frames and 3 other graphics; it is 700×48 pixels, 20 colors and the final size is 1600 bytes after TinyPNG compression.

It might worth to note that for character animation I used oldschool tricks of mirroring and rotating to reduce the frames. I have only drawn the standing and step with right foot frames and then “stand, step, stand, mirrored step” sequence. Also, I just rotated the “walking up” frames for the “walking right”, as well as the “walking down” for “walking left”. For pulling I did the same but used one frame only and played the sounds of steps.

The colors of customizable parts of the character (hard hat, shirt, pants) are replaced during render (and not on the sprite sheet) as each entry in the challenge leaderboard can be different.

Quick note: when opening an HTML from local disk and accessing its contents in JavaScript, like reading pixels of an image (rather than putting in DOM and showing to user, or using drawImage()) the Cross Origin protection kicks in and prevents the operation. Although when opening from a web server there is no such problem, this feature can be disabled but it’s strongly not recommended. Worth to note that JS engines track the data they handle so reading a canvas where a drawImage() copied a portion of a protected image will result in an error as well.

Sound effects

For the sound effect we used jsfxr to generate them.

I have added a few minor optimizations for jsfxr before compressing the source via Closure Compiler, and after that manually edited the compressed code to be more suitable for ZIP.

Music

The music was created entirely by my brother, Zsolt.

I have created a small tracker that implements the basic features of an XM, handles only one pattern and loops it, but also have a map that describes which channels should be played on which loop (and also an option for intro that will be only played once at the beginning) – this way a song can be made up of a short period of music but can be dynamic and changing on each loop. Also created a converter that reads an XM file and creates the JavaScript arrays for the player, so Zsolt could use MilkyTracker to compose.

The samples were produced using jsfxr just like the sound effects (they are actually in the same array) and are altered on the fly for player needs (changing playback speeds to create the notes).

I have also created a frontend for playback later, check it out.

Level design

The levels were by designed mostly by Zsolt, with a few exceptions.

They are stored as simple strings – ZIP is really good at compressing text.

The levels are locked by default. Only the already solved and the next two unsolved are unlocked, as an attempt to prevent the player from getting stuck on one level.

And as the steps are counted for each level it is just natural that achievement hunters appear, so we added an “A+” badge for them for completing the level with at most the minimal steps we completed them during development, displayed in the level selector.

Challenge mode

There is a challenge mode with online leaderboard showing the name, time and character profile picture for each record.

There are 6 challenges, each consist of 3-4 already completed levels, the player plays them one after the other and the time is recorded and shown on the leaderboard.

Statistics

The game collects statistics about each play: play time, moves made, pulls done, levels started, levels finished.

The statistics is stored on the server side, the games recevies the current snapshot of the statistics and the one from a minute before, and then interpolates them during the next minute – this makes the display much more active.

I loved to see the numbers rolling during the vote period, unfortunately the statistics are no longer available on the jam website. [Although I made a few screenshots, but I could not find them yet.]

Server side

The server keeps track of challenge leaderboards and statistics. It does not do anything more, really.

Compression

I basically go with the idea of writing readable code and leaving the low level optimization and compression to the tools.

Most of the time this works just fine as for a really simplified example…

if (x == 3)
{
    return true;
}

return false;

… is exactly…

return x == 3;

… but the former can be read more easily and the tool will rewrite it to the latter anyways.

For code compression I used Google Closure Compiler in two phases: first is the advanced optimization without whitespace removal, then the whitespace removal. This leaves a chance for (easier) manual/scripted optimization to be more suitable for the next tool (ZIP in this case).

The spritesheet is compressed using TinyPNG.

I used as few files in the compiled version as possible, as each file adds at least 130 bytes to the final ZIP file – this can be a lot and it increases with file name length.

For the final ZIP file the rules dictate to be a standard ZIP file, so no 7zip or other advanced formats are permitted.

To optimize the final ZIP I used AdvanceCOMP’s AdvZIP which basically recompresses an existing ZIP file but keeps compatibility.

About 90% of the build process – starting from readable code to entry ZIP – was automated in a build script. This way I could track the effect of each changes on the final size. The rest was manual editing of code just before Closure Compiler whitespace removal.

Fun fact: the last ZIP before the submitted one was 13313 bytes, 1 byte over the limit. I saved that byte by changing the player name input from “Player name:” to “New name:”.

Final thoughts

I think Bokosan is the most polished game I’ve created until today (alone or in a team), it has the most content and feels the most complete. It also got a lot of positive feedbacks and it was really fun to catch my collagues playing it, trying to get the “A+” badge on all levels.

However, the game has not ranked in the Top 10 in Desktop nor Mobile category, in Server category it finished at 6th place and in Facebook special on 3rd place.

A huge thank you to Andrzej Mazur for organizing js13kgames competition once again, Ray Larabie for the Jostix font, and Markus Neubrand for Jsfxr!

untitled13

In 2012 I found the js13kgames game development competition where the goal is to submit a HTML-CSS-Javascript game which is not more than 13 kB (13312 bytes) ZIPed, developed between 13th August and 13th September. It is also required to publish the uncompressed game code on GitHub – this way one can prove they created the game and more importantly: others can learn from it.

This last part was what got me, I was really curious how others fit games in such a small space – what later turned out to be much more than it looked on the first sight.

As the competition was already halfway or was over when I found it (I can’t really recall) I could only examine others’ entries and decide I will participate next year, in 2013. I also tried to gather some ideas and set up a concept before the competition start date – strictly not a line of code was written.

There is a theme each year (which is optional but worth bonus points if honored), for 2013 it was bad luck – and I skipped it entirerily…

Okay and now about my entry.

My concept was to create a 2 player networked tower defense/tower attack game.

I must admit I am really bad at naming things, I just called the main (and only) client side file 13312.js and not long before the deadline I came up with the exceptionally uncreative name untitled13.

Of course to fit in 13 kB entries usually do not use existing engines, they are written from scratch – a thing I don’t mind at all, I’ve been always curious about the details and liked the control one has when starts from the ground up.

For graphics and animations I decided to go with a fixed color palette and dynamically generated content. All the objects consist of one or more layers rendered by a function that takes a string that describes a series of commands and their parameters. It turned out only two commands were necessary, a “draw a closed polygon filled with a color” and a “draw an alpha overlay” for shadows. The parameters were simply X and Y coordinates of the points, represented as characters in the base64 character set (64 distinct values), multiplied by the size of sprite. (For example: sprite size is 64×32, string is “AAAzjj”, therefore the coordinates are 0;0, 0;25, 35;17).

The map consists of starting points for the attacker, paths for the units to walk on, path intersections with switches, a few objective tiles for crystals the defender has to defend and concrete blocks to build the towers upon. There are three types of monsters with different parameters (speed, health) and three towers with different parameters (attack type, damage).

The networking approach is pretty simple but the most interesting part: the initial state is known to both players (the map is fixed in the entry version, in later versions it is randomized but the seed is shared), the mechanics are known as well and the actions can be described easily. So I came up with the idea to run a whole game instance for both players, create delayed action queues for them and synchronize them. When one player clicks a button the action is added to their queue with the configured delay, sent to the other player’s game, and a bit later executed on both instances at the same moment.

To get this working I needed reliable advancement of simulation steps, one that is not affected by skipped frames. I created a tick() method that steps the simulation by exactly one time slice (one simulation step), so when a frame is requested by the browser I calculate how many tick should have happened up until then since the last frame and call tick() that many times. (I usually go with this method to keep games frame rate independent – skipped frames are less worse than game slowing down.)

As both players are running the same game (only playing different sides), are aware of every details (only hidden from them), all events are happening at the exact same time, the game should play out identically for both of them. Later I learned this is called simultaneous simulation and it is the core of great games like Age of Empires, Starcraft and a lot more.

In this form however there is a serious flaw. There is no guarantee that the actions will reach the other player before they are due, it is blindly trusted that it will. For example: Let the action queue delay be 10 frames. The defender clicks to build a tower at frame #100, this will be executed at frame #110. If the network lag is say 7 frames then the other player gets the notification at frame #107, still 3 frames before execution, the tower gets built at frame #110 in both instances and shoots the attacker just before it reaches the crystal. But if the network lag is more than the queue delay, say 13 frames then at frame #110 the tower gets built in one instance, then the notification arrives at frame #113 for the other player, tower gets built at frame #113 when it’s already too late, the crystal was reached and the game is over. In the defender’s instance the crystal is saved, in the attacker’s the crystal is destroyed – the game is out of sync.

To address this the other instance must be notified about no action as well periodically to make sure nothing is missed. And when there is no info, the game should pause and wait until it arrives. This is a feature I had no time to think through and implement before submitting my entry.

The server part is running in node.js (my very first application) and it is very basic, it just matches the players by their IDs and passes the messages between them.

Random minor thoughts: The map is a hexagonal grid, which made finding neighbour tiles for concrete block and fog calculation ugly, but it looks so much better than a square grid – although it is handled like that. The map has fog to hide tiles that are too far away from units/towers. The UI is usable but confusing and unhelpful at first.

The project is on GitHub, the version I submitted is in the js13kgames-2013-entry branch. I like to commit often (and to have descriptive commit messages) so the submitted version had 381 commits, but for some reason I have not uploaded the final ZIP file, it was around 11255 bytes of 13312, so I used up 84.5% of the limit.

The game was better received than I expected, it ranked 1st place in Server category and 9th place in Desktop category.

The competition had awesome prizes including various licenses for game engines and gamedev related services, ebooks, physical books and vouchers. It was organized by Andrzej Mazur who tirelessly did an epic job, I’d like to thank him again for organizing and hosting this event, finding sponsors, arranging prizes and of course organizing the event annually since then, so thank you Andrzej!

Check out js13kgames.com for previous and upcoming competitions.

101hero troubleshooting: skewed print

I’d like to cover the most common issues faced during first prints using a 101hero.

One of the most common is the skewed print, like in the following images.

Photo by Bogdan Bóg Anczarski posted in 101 Hero (Unofficial) Facebook Group.

This is most likely because of the stepper motors skipping steps. The problem can be traced back to one of two reasons: a belt touching the pylon or too high printing speed. Fortunately both problems can be fixed easily.

First check all your belts during a print. See if any of them is touching the plastic of the pylon, rubbing against it. If yes then check the pulley/carriage attached to the belt, find where the belt is fixed to it. Try to push the belt a little to the front or the back. If this does not help you can also try to loosen the screw holding the bearing in place on the top of the tower a bit.

If you see no problem with the belts it can be a problem with the stepper motors moving the belts. If the printing speed is too high then the motors tend to skip steps. The 101hero is not a fast printer, the stock motors support up to 14 mm/s printing and travelling. Unfortunately in some cases this speed is still too high for the motors even with stock motors and stock GCode files (from 101land.com).

If you have the model you should try to reduce the speed of the printing by configuring the printing as well as travelling speeds to no more than 14 mm/s, maybe down to 10 mm/s.

If you do not have the model but only the GCode file (the ones that can be downloaded from 101land.com) you need to add a command to reduce the printing speed to a certain percent. Open the “101hero” file on your SD card with a text editor, find the line starting with G28 (“Move to origin/home”), and add a line right after that stating “M220 S75”. This command is “Set speed factor override percentage” and the parameter is the percentage (14 mm/s * 75% = 10.5 mm/s).

If printing with these settings fixes the problem then you should try to use higher printing speeds (or higher percentage) to see what is the speed your printer is still reliable at.

If you still have the problem leave a comment here or post in the 101 Hero (Unofficial) Facebook group.

101hero: Assembling the printer

The first part covers the short story of the printer, the contents of the box, a few tips and warnings about the ways you might damage your printer unintentionally (if you have not read these you should).

This part is about assembling the printer step by step.

First connect only the PSU to the controller and switch it ON and OFF, the red LED should light up when it is ON.

Now connect the motors and end-stops to the controller (do not assemble anything yet) and attach the SD card slot. The intended order of the pylons as well as the motor-end-stop pairings are below:

After connecting these put a simple “go home” file on your SD card and then power it up. The motors should start moving the carriages (all three at the same time) until one hits its end-stop. Then the other two should stop and only that should move back and forth triggering the end-stop a few times. After this the remaining two should do the same one by one.

If all three carriages hit the end-stops then congrats, you have working motors and end-stops, this is a good start! If not… well then you are in the unlucky group of quite a few people, including me. In this case check out the 101 Hero (Unofficial) Facebook group.

You can restart the procedure by pushing the reset button or power cycling the controller board.

Now on to assembling the printer!

At the top of the pylon arrange the red-black cable of the end-stop so it goes through the shaft:

At the bottom of the pylon the red-black cable should go through the shaft just like above and arrange the cables of the motor like on the picture:

Make sure for each pylons that the spring on the belt is tight to provide tension…

… the belts are properly on the wheel at the top…

… and the bottom:

 

If the belts are off the wheels try to put them back carefully. If that is not enough then you can loosen the motor a bit by loosening the two screws holding it in place (3mm Hex Socket aka Allan heads). If you have to remove the belt tensioner spring here’s a video how to put it back.

Next, assemble the bottom plate and the three pylons (4 pcs of M4x12 self-tapping screws (PH1 head) for each pylon):

And then add the top plate (2 pcs of M4x12 self-tapping screws (PH1 head) for each pylon, NOTE: the two inner holes remain empty on each end on the top):

And now fix the effector to the rods. The facing of the head does not matter but it can be useful to see the fan (to see if it is working) and the door (for change filament) from the front. The first screw…

… the second screw (you can safely leave it hanging)…

… and the rest:

Now stick some painter’s tape to the glass plate (you don’t need to cover the whole glass just make sure you have no overlaps of the tape) put it on the bottom plate and fix it in place using the paper clips like this:

Now connect the rest of the cables to the controller box:

Next step is to load the filament. The extruder assembly has a door you can open, this door pushes the filament to the extruder motor which is supplying the filament to the heated nozzle. Open up the door and put the filament in at the top of the extruder assembly, guide it into the tube after the motor, like this:

When completed close the door and tighten the screws (2 pcs M3x12 self-tapping screws (PH1 head)).

After completing this step the printer should look like this:

Which is a mess. The cable management is pretty much non-existent for the 101hero by default. People use different cable organizing methods, the most popular is the spiral wrap but personally I like the zip tie method which is nothing special just zip tying the cables at a few centimeters interval, like this:

After completing this step you should cut the ties short, of course. One more thing I did was fixing the bunch of extruder cable to the top plate to an unused hole like this:

You need to leave about 25 cm (10 inches) of cable between the fixing point and the extruder assembly so the hot end can reach the farthest point safely.

And that’s all about assembling the printer. The next post will be about how to calibrate the 101hero.