BradLUG

June 12 2017:Guacamole, test-driven development, Ansible and Wireguard Meet

As no-one had prepared anything,

Brian mentioned that his search for a replacement for Tomboy had led him to Apache Guacamole which is currently an Apache Incubator project.

Darren introduced test driven development in which you start by writing a test to test the code and check that that works before you start writing the code. After you have written the code, you run the tests to check that it does what you want it to. Then you repeat the process for each new element of the code.

This led into a discussion of D, which builds on C++ but is intended to improve on it, and Haskell.

Oliver mentioned that he was doing a lot of his work with Ansible which allows him to manage Windows computers from Linux. Like SALT, it uses YAML and enables him to have everything in sync; there are also modules for routers and so on. He learned it through Linux Academy and stressed that you can start learning to use it with really simple things.

He mentioned that Aptana can convert Tomboy notes to Markdown.

Brian hadn’t been too impressed with Nextcloud; so we looked at Pydio, formerly AjaXplorer.

From this discussion Oliver mentioned Tmux, which allows you to run terminal simultaneously and receive notifications from terminals. It also remembers if a connection is dropped and reconnects seamlessly. John H added that it is recommended for command line updates to openSUSE.

We noted that Google is planning an ad-blocker for Chrome based on the recommendations of the Coalition for Betters Ads, a group of mainly advertisers frustrated that annoying and intrusive ads are giving them a bad name.

Oliver then shared a presentation about Wireguard, a new secure VPN tunnel to replace OpenVPN and IPsec. It uses UDP, is configured using wg or wg-quick and existing resources such as ip and iptables. Configuration files can be stored in /etc/wireguard/vpn0.conf while static public keys provide the identities and, unless an authenticated connection is made, the endpoints remain invisible. The software must be on both endpoints.

We then turned to Google’s plans to make it easier for users to update Android ’phones by reconfiguring the software. John H commented that he had been astonished at this because Gary Kildall had created CP/M in the 1970s to enable hardware manufacturers to customise the hooks on the hardware side and software developers to be assured of a consistent interface whatever the hardware on which CP/M was installed.

We finished by discussing ways of running git including Cgit and Gitea.

May 8 2017: Intel AMT, EdgeXFactory, Shodan and CiviCRM Meet

We started by discussing the Intel AMT bug which had emerged recently, allowing empty password access to a remote system. Alice pointed out that AMT has to be enabled and is normally only enabled in enterprise computers; so devices using AMD chips and the vast majority of devices using Intel chips will not have been affected by it.

Brian then introduced us to the EdgeXFoundry which is based on the Dell FUSE code and aims to improve interoperability between IoT devices. However, there is little to show so far on the EdgeXFoundry website! They then shared with us a recent Linux News podcast.

Alice introduced us to the wonders of Shodan, a search engine for IoT devices, with which they showed us the distribution of a variety of devices worldwide.

They also shared PonySay which displays a pony in your terminal and, in response to a question about Kafka in Docker, explained that Kafka in Docker provides for messaging between things.

John H rounded off the evening with a quick demonstration of CiviCRM, Customer Relations Management software aimed at voluntary (‘non-profit’) organisations. Unlike commercial CRM offerings, it runs on top of Drupal, Django or Wordpress and is modular, allowing organisations to install only the modules they need.

It uses a mysql database to hold contacts and relationships between them. The default contacts are individual, household and organisation but these can be extended to suit the organisation. Relationships can then be set up between contacts; for example, an individual can have the relationship of officer within an organisation.

Users sign in to the underlying website software and those with the relevant permissions are then permitted to progress to CiviCRM and exercise these permissions within CiviCRM. Because permissions can be linked to particular relationships, someone with an officer relationship can have specific permissions related to a specific group of individuals in a specific part of an organisation. This enables organisations to comply fully with the General Data Protection Regulation.

Among the modules are ones for Membership, Contributions (that is, payments, which must be linked to some accounting software), Mailings (to send out bulk emails that comply with opt-out and unsubscribe requirements), Reports, Campaigns, Grants (for organisations that give out money) and Clients (for organisations which have a clientele).

April 10 2017: Creating a Git Repository Meet

We welcomed Ben, a Python programmer from Cambridge, who was on a working trip to West Yorkshire.

Brian asked about notetaking apps because Tomboy was no longer synchronising properly. He would prefer a web-based app and had looked at Minimatch which uses NPM.

This provoked a discussion about developers dropping features.

Then, while David S led a private discussion at one end of the room,

Alice demonstrated creating a local Git repository for the BradLUG website. Though John had been able to create such a repository at the earlier meeting, now that all the website content had been added, his installation was now missing a lot of dependencies.

In order to make the Github implementation of Jekyll work on his machine, John had to install ruby-devel, gcc, zlib, libxml and nodejs. Then running as root

gem install bundle

installed everything else that was needed.

Next, John had to log in to Github and create a fork of the site, providing an SSH key for his fork.

Then, running as user on the local machine, he entered:

git clone git@github.com:john-hudson/bradlug.github.io.git
cd bradlug.github.io/
bundle.ruby2.1
bundle.ruby2.1 exec rake preview

to create the local repository and allow its contents to be viewed by entering 127.0.0.1:4000 in a browser.

Alice then helped John to modify the last meeting notes on the website to remove the surplus lines created during the transfer from the old site and create the report of the March meeting in 2017-03-19-march-13-2017-farewell-to-stephane-meet.markdown which can be used as a template for future posts. (As John had prepared the notes in HTML, it was noted that HTML in Markdown has to be continuous with no spaces or carriage returns.)

Thereafter, it was a matter of issuing the commands to add the March meeting notes to the local repository and then commit them to John’s Github repository before going online to merge them.

Alice drew people’s attention to the Leeds Digital Festival which is taking place between April 22nd and April 29th.

March 13 2017: Farewell to Stephane Meet

Only Stéphane had announced something to share; so

John W asked about freezing rows and columns in LibreOffice Calc. This has changed recently but involves placing the cursor in the highest cell on the left hand side which you do not want to freeze and then selecting Windows->Freeze in older versions and View->Freeze cells in the newer versions.

John H commented that he remembered having this feature in Supercalc in the 1980s and, in response to a question about Excel, said that, as with Gnumeric today, Supercalc was aimed more at scientific and engineering work than business. It also had stunning vector graphics charts, originally available as a separate program with the CP/M version and then incorporated into the DOS version.

John had encountered a problem accessing Windows on his laptop after he had installed a Debian mini-iso on a flash drive. After various attempts to edit Grub, including using the Super Grub Disk program, System rescue revealed that the Windows partition tables had been trashed, presumably during the mini-iso installation.

Stéphane’s big news was that, after being sacked following Brexit and trying to make his way as a freelance consultant with some success but not with sufficient income to prevent his reserves draining away, he had begun to apply for a variety of Java related jobs, in one instance not succeeding because he was overqualified, and then heard that he had a new landlord who was terminating all contracts. He had been almost at the end of applying for a job which involved a variety of online tests which he had managed to complete when he got the offer of a consultant post back in Paris, in effect with his old employer. So he will be returning to Paris after ten years in the UK.

For his final contribution, he demonstrated using screen to write and run a small program.

Au revoir, Stéphane; nous vous envoyons nos meilleures pensées.

February 13 2017: American Fuzzy Lop, SlackBuilds, LowEndSpirit, GPL violations and Jitsi Meet

As only David had come with anything to share, we rambled round a wide range of topics.

Nick, who was with us for the first time since 2015, showed us the ThinkPad he had bought for £80 on eBay and told us that he had moved on from SkyBet to Leeds University Department of Engineering where there is a lot of Linux, mostly CentOS and using Puppet, and a wide range of computing resources up to an HPC cluster which is used by, among others, the European Space Agency.

He asked whether anyone had any thoughts on using btrfs at which David went apoplectic while John said that SUSE Enterprise had been using it for some time apparently without problems but, when they had introduced it to openSUSE just over a year ago, a lot of people had found it filled up their root partition suggesting that it is best used by those who know how to use it.

David shared the paper by Vegard Nossum and Quentin Casasnovas on applying American Fuzzy Lop to filesystems given at Vault 2016. This showed that it had taken only five seconds to encounter a bug in btrfs but two hours in Ext4.

David then introduced the Repology website which tracks the versions of packages in the various repositories along with the maintainers.

He next updated us on how he had decided to automate kernel updates in Slackbuilds. He had settled on using curl, grep, sed and grep to search for the relevant words in the kernel.org RSS feed, save the results in releases.new and then compare that with releases.csv. If there are any changes, releases.new is renamed releases.csv and the building commences for the four kernels, 32bit and 64bit for the stable and the current (beta) versions. At the moment, this is all in a box running 24/7 because he has not yet set up the wake up LAN.

Darren introduced us to the LowEndSpirit VPS. This runs on Solus VM which uses, among others, OpenVZ which is commercially supported by Virtuozzo.

David then asked us about Patrick McHardy. The background is that the Software Freedom Conservancy and the Free Software Foundation published guidelines on GPL enforcement principles. There is GPL Violations, a German organisation founded in 2004, which sometimes pursues violators through the courts but Patrick McHardy has been pursuing violators in complete disregard for the principles and has had all his kernel commit rights taken away.

We then looked at the sketch on Youtube about Sean Spicer that had appeared on Saturday Night Live in the US.

John rounded off the evening by linking up with David using the browser version of Jitsi Meet, one of the options Mozilla recommends following the withdrawal of Firefox Hello.

January 9 2017: TU100, automatic static website creation and Slackbuilds

Darren shared some of the problems which had appeared on the Open University TU100 My digital life forums relating to the SenseSense programming language which the Open University have developed from Scratch for use with mature students. Darren himself had had a problem because his 64-bit OS was just that; it had no 32-bit libraries.

Kriss and Shi Then shared their experiences of automating the development of static websites. Their first attempt was a ‘bootstrap’ CMS using App Engine and text files as input to generate the code which was tried out among the gaming community.

But they decided to step away from this in part because App Engine is aimed at large scale installations.

Their second attempt was ‘pancake’ using Open Resty which runs LuaJIT in Nginx to take advantage of Nginx’s efficiency. It was modular, used chunks and allowed multiple CSS files to enable different layouts within a website.

From this they moved on to ‘pated’ which has some similarities with Jekyll and decided to use the hat character as part of their naming convention; a directory starts with a hat character and file has one somewhere in its name. They illustrated that you can do quite a bit with static websites by demonstrating one showing aid data.

In practice, organising the data is the hardest part; the chunks are all in JSON format and you could use those on a dynamic website.

David C then gave us a running commentary on automating kernel security patches (of which there has recently been a rash) at Slackbuilds.

The idea is to have a single system that will notice patches to the stable kernel and apply them to Slackbuilds in such a way that builds will be reproducible, that is, however many people build it, all builds will be bit identical.

It needs something to read the RSS feed and apply awk and grep — in other words, a BASH script! This needs to prompt the router to wake up in order to start building the new kernel.

As he wants something simple, [If This Then That]](https://ifttt.com/) and Lighttpd on the router would seem to do the trick, though it was suggested he might want to consider Huginn instead of IFTTT.

This led into a discussion of the possibility of using a Twitter feed rather than RSS, in which case you would need to use https: with a single referer, and the availability of really cheap web servers.

December 12 2016: PIC micro-controller, Ham radio logging and LXQt

Roger who hails from South Yorkshire and had stopped by on a return journey from Sutton Bank shared his experiences of using Linux with the PICkit and PIC microcontroller.

He had built his first computer in the 1970s, worked on a PDP-12 and programmed in Fortran, assembler, C+ and C++ mostly in the Steel Industry. Since he had retired he had been trying to show that you can do everything in Linux using GTK+ for which he had found formal descriptions but not always good examples. So he had decided to write his own examples.

As his hobby is gliding a lot of his examples relate to gliding. One is an anemometer head with optical readout which uses the PIC’s digital ports. He uses the Small Device C Compiler (SDCC) which is supported in PIC along with the GNU utilities.

You need to pay €20 for the PIC programmer because you cannot inspect the internals of a microchip or step through the C code line by line; so you need the simulator.

The PIC chip is a RISC chip and normally takes four cycles to execute a function; it also has analogue ports. Obviously, you have to understand the chip to decide what you want to do with it.

This led into a discussion of the merits of the PIC and Atmel chips.

Roger went on to say that he had mostly done the backend at work; now he was doing the frontend to prove the concept that it could also be done in Linux. He had completed three projects to illustrate this:

1. a device to send signals back to the winch during glider takeoff

2. the anemometer head

3. a battery charger which also calculated the capacity of the lead-acid batteries used in gliders — his most complicated project to date.

Stephane said that, since getting his amateur radio licence, he had been working at 20MHz HF and 70m DMR and built his own antenna which could pick up signals from Leeds in Harrogate. He has made contacts with Australia, Canada, the US, Germany and France.

A ham contact is called a QSO and hams like to log them; among the options are:

• a paper log

• a spreadsheet

• Log40M

• (online) qr2.com

but most logging software

• is Windows based

• lacks lifespan maintenance

• is cluttered and complex with screens filled with buttons.

He wanted to:

• understand all the possibilities of a log

• sharpen his coding skills as Java 8 has added functions and predicates to its object oriented framework

• make it his own.

Among possible formats for a log are

• Cabrillo

• ADIF (Amateur Data Interchange Format) which is effectively the standard.

Using xml data and xml formatting it is practical to build logs to a specific format and the ADIF 3.4 documentation is all available.

So he decided to use a client-server model which also helps in competitions when each contestant can have a client and to use xml validation in both directions.

He built it using JDK, Eclipse and the Postman plugin for Chrome.

The server loads validated data and stores it in memory; the client sends one or more QSOs embedded in an ADS object within validated xml to the server and can request QSOs from the server in the same format or query the server database.

In response to questions about the database, Stephane said the database consisted of QSOs stored according to an xml format from which it was possible to generate Java objects. He had tried to make it as small as possible using the built-in Java server. One could hold the data in an enterprise server.

Java takes a chunk of RAM and allocates space for the database which can be loaded and dumped. It can also be validated in Java.

The xml schema is contained in an xsd file and the xml config file will have details of the log books and the specific services being used.

The REST API offered post and query and he had created a third option of ‘get last n QSOs.’

The client page is simply an HTML page running on node.js with a script pointing to the Java code. He used the W3C Chocolate CSS stylesheet. The client retrieves data from the server every three seconds.

So the server uses HTML and Javascript and is CORS enabled. The client relies on an xml schema and config file.

Q&A; In response to a question about the server interface, he demonstrated some of the available commands — ca, show, freq — and how it would identify false entries. It also reports a successful log.

Asked about how time was recorded, he said there were options to enter time on and time off and to enter the time automatically, each of which could be edited to take account of time lags between an event on the radio and the entry in the log.

In answer to a question about creating a GUI, he said that he wanted the CLI for auto-completion and many radio amateurs were used to DOS style interfaces. He was thinking about adding a query feature. The server has 1,800 lines of code, the client a few hundred.

Asked about using JSON, he said that JSON is good for simple data and there are lots of JSON implementations which people who prefer JSON can use but he prefers xml. It would certainly be possible to put the request in xml and the response in JSON.

In response to a question about someone pulling the plug, Stephane said he had not taken account of that but could add a feature to perform dumps at set intervals to preserve the data.

John H then explained by way of introduction that the laptop he was using was his travelling laptop on which he had installed openSUSE LEAP 42.1 but had found the video unstable; some times it would run fine for hours; other times he had to reboot several times before it became stable. However, the USB hub on his main computer had decided it no longer wanted to operate bidirectionally and so he had moved everything on to this computer.

Then he found that btrfs had taken up all the space on his root partition; so he had installed LEAP 42.2 with an Ext4 root partition and begun to play with the LXQt desktop, a joint venture between some of the LXDE developers and the razor-qt developers which was in effect a Qt equivalent to Xfce rather than an alternative to LXDE.

He had found this very stable with KDE applications which had crashed in Plasma running smoothly though DigiKam, the KDE photo-editing program did not like it.

Further details are available from John’s website.

November 14 2016: Configuration management

David S did a presentation on configuration management or how to make sure that everything you need is set up as you want it to be whether on one or on a thousand devices.

As of 2016 there are four actively supported free and open source configuration management programs: Ansible, supported by RedHat, Puppet, Chef and SALT. Each have their quirks and idiosyncrasies.

SALT uses Python modules which are extensible along with ZeroMQ for messaging. Scripts are written in YAML with Jinja2 providing functions. ZeroMQ has a temporary buffer in which to hold messages if there is a break in communication.

SALT is less impressive at scale but it can use SSH instead of ZeroMQ where there is no agent on a device whereas Chef, for example, requires an agent on every device. However SSH is not fault tolerant.

A master device holds all the config files which determine how minions are to be configured. Note that minions cannot see other minions. The target specifies which minions are to be configured; the state specifies how they are to be configured and the configuration is undertaken by Python modules.

A grain describes the configuration of the particular device on which it resides whereas a pillar holds the configuration for a minion on the master. A file server on the master holds the files which a master will send to a minion.

Setting up is primarily a matter of exchanging keys. The YAML files primarily contain data structures but YAML is sensitive to indentation while Jinja2 handles commands.

SALT has its own command line options and can be run on Windows by using Chocolatey.

David noted that configuration management needs to be see as part of a wider picture which includes provisioning, the Cloud, containers, monitoring and workflow in which each program has its strengths and weaknesses.

He concluded with a summary of what SALT is not very good at, quite good at and very good at.

John H raised a problem of sub-domains on a website disappearing from time to time following a move to Cloudflare and the consensus was that the problem would be a badly configured nginx config file.

October 10 2016: Manchester BarCamp and tracking intrusions on uWSGI

As no-one had prepared anything specially for the meeting and David S was occupied trying to get Adobe Flash to work on John W’s computer, we chatted among ourselves with Brian and Ash sharing their experiences of Manchester BarCamp. The arrangements had been better this year with half a dozen lecture rooms available. Brian had given his IoT talk which he had tried out on us the previous month and they had enjoyed sessions on Hacker Packet Radio and Git.

David S then managed to share with us an episode at work when a colleague had alerted him to a message via Sentry.

`InvalidSchema: No connection adapters were found for 'file:///etc/passwd'`

Because they use Apache with a reverse proxy, namely, uWSGI, David had to trawl through the uWSGI server’s logs, finding a wide range of attempts to crack the server leading up to the attempt which had prompted the warning at 10.40 am on 3 October.

A look in AbuseIPDB showed that the IP address from which these attempts were being made was in Ukraine though the actual attacked could have been in another country and it appeared they were using the acunetix web scanner.

As one of their customers had been involved in an anti-bribery initiative in Ukraine, one line of enquiry was that this might be an attempt at a revenge attack. However, analysis of the logs showed that servers related to other customers, in one particular data centr’’s netblock, had previously been gently probed by the same netblock in Ukraine, suggesting that the attack wasn’t targeted at the specific customer but just a coincidence.

David had ended up editing IP tables as Fail2ban will not work with the uWSGI logs. [The is a more detailed account of this incident on the Idelmoor Technical Blog.]

This led into a discussion of how the organisation’s servers are managed and David said they used SALT partly because they can do everything using SSL. SALT uses ZeroMQ for messaging and YAML for scripting.

After this brief introduction, David was press-ganged into doing a more detailed presentation next month.

September 12 2016: MQTT, Node-RED, micro-benchmarks and review of the year

Brian gave a demonstration of live messaging between ‘things’ using MQTT in which members were encouraged to participate; this involved installing Mosquitto, a message ‘broker’ for MQTT, and then connecting to the temporary wi-fi network which Brian had set up.

Brian then moved on to demonstrating Node-RED, a Scratch like graphical interface for controlling the flow of messages between ‘things’ by software switching which can be installed on a Raspberry Pi. It supports a wide range of messaging formats including Asterisk, UDP and Twitter. Once configured, the interface can be switched off.

Brian warned about the need to deal with ‘switch bounce’ which occurs because, in for example a 5V circuit, Logic 0 is generally associated with less than 0.8V and Logic 1 with more than 2.4V making the voltages between 0.8V and 2.4V carry uncertain values. Ideally one should use a hardware latch to stop the effects of ‘bounce’ but a software latch could also be used.

In the course of this very successful demonstration, documented here, David S and Nigel managed to write a script to reconfigure the flows in the Node-RED page using Mosquitto as Brian had not yet worked out how to secure the messaging flow [but has done so since!].

Matt then presented ‘Tales of Linux micro-benchmarks.’ While benchmarks are intended to measure overall performance, micro-benchmarks are intended to measure the effects of very small code changes rather than overall performance. If you look on the Internet, you will find plenty of conflicting views about them.

So he gave some examples of working with them. Siege is supposed to measure the response of a server to increasing load. However, the code includes a line which means that, in effect, it is measuring itself at the same time as it is measuring the software.

Hackbench, a scheduler benchmark, needs to be run on a limited number of instances. In one case, Matt had found that 70% of the measurement related to setting up the system and only 30% to the software being measured because of the number of instances being measured.

Pipetest, which tests event delivery, produces some very pretty graphs but, when you look at the resolution of the graphs, it is so high that it is unhelpful. The resolution needs to be set to something useful.

So pick your micro-benchmark carefully and take care over the calls you are using. Make sure you understand what the data actually means.

Q&A; When might you use them? Mostly for parts of the system to which you do not have access or where you don’t want other parts of the system to influence the measurement.

Are they primarily to enable developers to identify regressions? That and also for capacity planning.

John H rounded off the evening with a quick review of the year.

Ways to get in touch

Mailing List
The mailing list is kept up to date with all the details about the meetings, socials, ideas, and questions.
BradLUG at mailman.lug.org

Twitter
http://twitter.com/bradlug