Welcome to the Bradford GNU/Linux site; we are a self-help group for GNU/Linux, Open Source and Free Software users in and around the Bradford District, West Yorkshire
Why not come along to a meeting.
As no-one had prepared anything,
Darren introduced test driven development in which you start by writing a test to test the code and check that that works before you start writing the code. After you have written the code, you run the tests to check that it does what you want it to. Then you repeat the process for each new element of the code.
Oliver mentioned that he was doing a lot of his work with Ansible which allows him to manage Windows computers from Linux. Like SALT, it uses YAML and enables him to have everything in sync; there are also modules for routers and so on. He learned it through Linux Academy and stressed that you can start learning to use it with really simple things.
From this discussion Oliver mentioned Tmux, which allows you to run terminal simultaneously and receive notifications from terminals. It also remembers if a connection is dropped and reconnects seamlessly. John H added that it is recommended for command line updates to openSUSE.
We noted that Google is planning an ad-blocker for Chrome based on the recommendations of the Coalition for Betters Ads, a group of mainly advertisers frustrated that annoying and intrusive ads are giving them a bad name.
Oliver then shared a presentation about Wireguard, a new secure VPN tunnel to replace OpenVPN and IPsec. It uses UDP, is configured using
wg-quick and existing resources such as
iptables. Configuration files can be stored in
/etc/wireguard/vpn0.conf while static public keys provide the identities and, unless an authenticated connection is made, the endpoints remain invisible. The software must be on both endpoints.
We then turned to Google’s plans to make it easier for users to update Android ’phones by reconfiguring the software. John H commented that he had been astonished at this because Gary Kildall had created CP/M in the 1970s to enable hardware manufacturers to customise the hooks on the hardware side and software developers to be assured of a consistent interface whatever the hardware on which CP/M was installed.
We started by discussing the Intel AMT bug which had emerged recently, allowing empty password access to a remote system. Alice pointed out that AMT has to be enabled and is normally only enabled in enterprise computers; so devices using AMD chips and the vast majority of devices using Intel chips will not have been affected by it.
Brian then introduced us to the EdgeXFoundry which is based on the Dell FUSE code and aims to improve interoperability between IoT devices. However, there is little to show so far on the EdgeXFoundry website! They then shared with us a recent Linux News podcast.
Alice introduced us to the wonders of Shodan, a search engine for IoT devices, with which they showed us the distribution of a variety of devices worldwide.
John H rounded off the evening with a quick demonstration of CiviCRM, Customer Relations Management software aimed at voluntary (‘non-profit’) organisations. Unlike commercial CRM offerings, it runs on top of Drupal, Django or Wordpress and is modular, allowing organisations to install only the modules they need.
It uses a mysql database to hold contacts and relationships between them. The default contacts are individual, household and organisation but these can be extended to suit the organisation. Relationships can then be set up between contacts; for example, an individual can have the relationship of officer within an organisation.
Users sign in to the underlying website software and those with the relevant permissions are then permitted to progress to CiviCRM and exercise these permissions within CiviCRM. Because permissions can be linked to particular relationships, someone with an officer relationship can have specific permissions related to a specific group of individuals in a specific part of an organisation. This enables organisations to comply fully with the General Data Protection Regulation.
Among the modules are ones for Membership, Contributions (that is, payments, which must be linked to some accounting software), Mailings (to send out bulk emails that comply with opt-out and unsubscribe requirements), Reports, Campaigns, Grants (for organisations that give out money) and Clients (for organisations which have a clientele).
We welcomed Ben, a Python programmer from Cambridge, who was on a working trip to West Yorkshire.
This provoked a discussion about developers dropping features.
Then, while David S led a private discussion at one end of the room,
Alice demonstrated creating a local Git repository for the BradLUG website. Though John had been able to create such a repository at the earlier meeting, now that all the website content had been added, his installation was now missing a lot of dependencies.
In order to make the Github implementation of Jekyll work on his machine, John had to install
nodejs. Then running as root
gem install bundle
installed everything else that was needed.
Next, John had to log in to Github and create a fork of the site, providing an SSH key for his fork.
Then, running as user on the local machine, he entered:
git clone firstname.lastname@example.org:john-hudson/bradlug.github.io.git
bundle.ruby2.1 exec rake preview
to create the local repository and allow its contents to be viewed by entering
127.0.0.1:4000 in a browser.
Alice then helped John to modify the last meeting notes on the website to remove the surplus lines created during the transfer from the old site and create the report of the March meeting in 2017-03-19-march-13-2017-farewell-to-stephane-meet.markdown which can be used as a template for future posts. (As John had prepared the notes in HTML, it was noted that HTML in Markdown has to be continuous with no spaces or carriage returns.)
Thereafter, it was a matter of issuing the commands to add the March meeting notes to the local repository and then commit them to John’s Github repository before going online to merge them.
Alice drew people’s attention to the Leeds Digital Festival which is taking place between and .
Only Stéphane had announced something to share; so
John W asked about freezing rows and columns in LibreOffice Calc. This has changed recently but involves placing the cursor in the highest cell on the left hand side which you do not want to freeze and then selecting Windows->Freeze in older versions and View->Freeze cells in the newer versions.
John H commented that he remembered having this feature in Supercalc in the 1980s and, in response to a question about Excel, said that, as with Gnumeric today, Supercalc was aimed more at scientific and engineering work than business. It also had stunning vector graphics charts, originally available as a separate program with the CP/M version and then incorporated into the DOS version.
John had encountered a problem accessing Windows on his laptop after he had installed a Debian mini-iso on a flash drive. After various attempts to edit Grub, including using the Super Grub Disk program, System rescue revealed that the Windows partition tables had been trashed, presumably during the mini-iso installation.
Stéphane’s big news was that, after being sacked following Brexit and trying to make his way as a freelance consultant with some success but not with sufficient income to prevent his reserves draining away, he had begun to apply for a variety of Java related jobs, in one instance not succeeding because he was overqualified, and then heard that he had a new landlord who was terminating all contracts. He had been almost at the end of applying for a job which involved a variety of online tests which he had managed to complete when he got the offer of a consultant post back in Paris, in effect with his old employer. So he will be returning to Paris after ten years in the UK.
For his final contribution, he demonstrated using screen to write and run a small program.
Au revoir, Stéphane; nous vous envoyons nos meilleures pensées.
As only David had come with anything to share, we rambled round a wide range of topics.
Nick, who was with us for the first time since 2015, showed us the ThinkPad he had bought for £80 on eBay and told us that he had moved on from SkyBet to Leeds University Department of Engineering where there is a lot of Linux, mostly CentOS and using Puppet, and a wide range of computing resources up to an HPC cluster which is used by, among others, the European Space Agency.
He asked whether anyone had any thoughts on using
btrfs at which David went apoplectic while John said that SUSE Enterprise had been using it for some time apparently without problems but, when they had introduced it to openSUSE just over a year ago, a lot of people had found it filled up their root partition suggesting that it is best used by those who know how to use it.
David shared the paper by Vegard Nossum and Quentin Casasnovas on applying American Fuzzy Lop to filesystems given at Vault 2016. This showed that it had taken only five seconds to encounter a bug in
btrfs but two hours in
David then introduced the Repology website which tracks the versions of packages in the various repositories along with the maintainers.
He next updated us on how he had decided to automate kernel updates in Slackbuilds. He had settled on using
grep to search for the relevant words in the kernel.org RSS feed, save the results in
releases.new and then compare that with
releases.csv. If there are any changes,
releases.new is renamed
releases.csv and the building commences for the four kernels, 32bit and 64bit for the stable and the current (beta) versions. At the moment, this is all in a box running 24/7 because he has not yet set up the wake up LAN.
David then asked us about Patrick McHardy. The background is that the Software Freedom Conservancy and the Free Software Foundation published guidelines on GPL enforcement principles. There is GPL Violations, a German organisation founded in 2004, which sometimes pursues violators through the courts but Patrick McHardy has been pursuing violators in complete disregard for the principles and has had all his kernel commit rights taken away.
We then looked at the sketch on Youtube about Sean Spicer that had appeared on Saturday Night Live in the US.
Darren shared some of the problems which had appeared on the Open University TU100 My digital life forums relating to the SenseSense programming language which the Open University have developed from Scratch for use with mature students. Darren himself had had a problem because his 64-bit OS was just that; it had no 32-bit libraries.
Kriss and Shi Then shared their experiences of automating the development of static websites. Their first attempt was a ‘bootstrap’ CMS using App Engine and text files as input to generate the code which was tried out among the gaming community.
But they decided to step away from this in part because App Engine is aimed at large scale installations.
Their second attempt was ‘pancake’ using Open Resty which runs LuaJIT in Nginx to take advantage of Nginx’s efficiency. It was modular, used chunks and allowed multiple CSS files to enable different layouts within a website.
From this they moved on to ‘pated’ which has some similarities with Jekyll and decided to use the hat character as part of their naming convention; a directory starts with a hat character and file has one somewhere in its name. They illustrated that you can do quite a bit with static websites by demonstrating one showing aid data.
In practice, organising the data is the hardest part; the chunks are all in JSON format and you could use those on a dynamic website.
David C then gave us a running commentary on automating kernel security patches (of which there has recently been a rash) at Slackbuilds.
The idea is to have a single system that will notice patches to the stable kernel and apply them to Slackbuilds in such a way that builds will be reproducible, that is, however many people build it, all builds will be bit identical.
It needs something to read the RSS feed and apply
grep — in other words, a BASH script! This needs to prompt the router to wake up in order to start building the new kernel.
As he wants something simple, [If This Then That]](https://ifttt.com/) and Lighttpd on the router would seem to do the trick, though it was suggested he might want to consider Huginn instead of IFTTT.
This led into a discussion of the possibility of using a Twitter feed rather than RSS, in which case you would need to use https: with a single referer, and the availability of really cheap web servers.
Roger who hails from South Yorkshire and had stopped by on a return journey from Sutton Bank shared his experiences of using Linux with the PICkit and PIC microcontroller.
He had built his first computer in the 1970s, worked on a PDP-12 and programmed in Fortran, assembler, C+ and C++ mostly in the Steel Industry. Since he had retired he had been trying to show that you can do everything in Linux using GTK+ for which he had found formal descriptions but not always good examples. So he had decided to write his own examples.
As his hobby is gliding a lot of his examples relate to gliding. One is an anemometer head with optical readout which uses the PIC’s digital ports. He uses the Small Device C Compiler (SDCC) which is supported in PIC along with the GNU utilities.
You need to pay €20 for the PIC programmer because you cannot inspect the internals of a microchip or step through the C code line by line; so you need the simulator.
The PIC chip is a RISC chip and normally takes four cycles to execute a function; it also has analogue ports. Obviously, you have to understand the chip to decide what you want to do with it.
This led into a discussion of the merits of the PIC and Atmel chips.
Roger went on to say that he had mostly done the backend at work; now he was doing the frontend to prove the concept that it could also be done in Linux. He had completed three projects to illustrate this:
a device to send signals back to the winch during glider takeoff
the anemometer head
a battery charger which also calculated the capacity of the lead-acid batteries used in gliders — his most complicated project to date.
Stephane said that, since getting his amateur radio licence, he had been working at 20MHz HF and 70m DMR and built his own antenna which could pick up signals from Leeds in Harrogate. He has made contacts with Australia, Canada, the US, Germany and France.
A ham contact is called a QSO and hams like to log them; among the options are:
but most logging software
is Windows based
lacks lifespan maintenance
is cluttered and complex with screens filled with buttons.
He wanted to:
understand all the possibilities of a log
sharpen his coding skills as Java 8 has added functions and predicates to its object oriented framework
make it his own.
Among possible formats for a log are
Using xml data and xml formatting it is practical to build logs to a specific format and the ADIF 3.4 documentation is all available.
So he decided to use a client-server model which also helps in competitions when each contestant can have a client and to use xml validation in both directions.
He built it using JDK, Eclipse and the Postman plugin for Chrome.
The server loads validated data and stores it in memory; the client sends one or more QSOs embedded in an ADS object within validated xml to the server and can request QSOs from the server in the same format or query the server database.
In response to questions about the database, Stephane said the database consisted of QSOs stored according to an xml format from which it was possible to generate Java objects. He had tried to make it as small as possible using the built-in Java server. One could hold the data in an enterprise server.
Java takes a chunk of RAM and allocates space for the database which can be loaded and dumped. It can also be validated in Java.
The xml schema is contained in an xsd file and the xml config file will have details of the log books and the specific services being used.
The REST API offered post and query and he had created a third option of ‘get last n QSOs.’
The client page is simply an HTML page running on node.js with a script pointing to the Java code. He used the W3C Chocolate CSS stylesheet. The client retrieves data from the server every three seconds.
Q&A; In response to a question about the server interface, he demonstrated some of the available commands —
freq — and how it would identify false entries. It also reports a successful log.
Asked about how time was recorded, he said there were options to enter time on and time off and to enter the time automatically, each of which could be edited to take account of time lags between an event on the radio and the entry in the log.
In answer to a question about creating a GUI, he said that he wanted the CLI for auto-completion and many radio amateurs were used to DOS style interfaces. He was thinking about adding a query feature. The server has 1,800 lines of code, the client a few hundred.
Asked about using JSON, he said that JSON is good for simple data and there are lots of JSON implementations which people who prefer JSON can use but he prefers xml. It would certainly be possible to put the request in xml and the response in JSON.
In response to a question about someone pulling the plug, Stephane said he had not taken account of that but could add a feature to perform dumps at set intervals to preserve the data.
John H then explained by way of introduction that the laptop he was using was his travelling laptop on which he had installed openSUSE LEAP 42.1 but had found the video unstable; some times it would run fine for hours; other times he had to reboot several times before it became stable. However, the USB hub on his main computer had decided it no longer wanted to operate bidirectionally and so he had moved everything on to this computer.
Then he found that
btrfs had taken up all the space on his root partition; so he had installed LEAP 42.2 with an
Ext4 root partition and begun to play with the LXQt desktop, a joint venture between some of the LXDE developers and the razor-qt developers which was in effect a Qt equivalent to Xfce rather than an alternative to LXDE.
He had found this very stable with KDE applications which had crashed in Plasma running smoothly though DigiKam, the KDE photo-editing program did not like it.
Further details are available from John’s website.
David S did a presentation on configuration management or how to make sure that everything you need is set up as you want it to be whether on one or on a thousand devices.
SALT uses Python modules which are extensible along with ZeroMQ for messaging. Scripts are written in YAML with Jinja2 providing functions. ZeroMQ has a temporary buffer in which to hold messages if there is a break in communication.
SALT is less impressive at scale but it can use SSH instead of ZeroMQ where there is no agent on a device whereas Chef, for example, requires an agent on every device. However SSH is not fault tolerant.
A master device holds all the config files which determine how minions are to be configured. Note that minions cannot see other minions. The target specifies which minions are to be configured; the state specifies how they are to be configured and the configuration is undertaken by Python modules.
A grain describes the configuration of the particular device on which it resides whereas a pillar holds the configuration for a minion on the master. A file server on the master holds the files which a master will send to a minion.
Setting up is primarily a matter of exchanging keys. The YAML files primarily contain data structures but YAML is sensitive to indentation while Jinja2 handles commands.
SALT has its own command line options and can be run on Windows by using Chocolatey.
David noted that configuration management needs to be see as part of a wider picture which includes provisioning, the Cloud, containers, monitoring and workflow in which each program has its strengths and weaknesses.
He concluded with a summary of what SALT is not very good at, quite good at and very good at.
John H raised a problem of sub-domains on a website disappearing from time to time following a move to Cloudflare and the consensus was that the problem would be a badly configured
nginx config file.
As no-one had prepared anything specially for the meeting and David S was occupied trying to get Adobe Flash to work on John W’s computer, we chatted among ourselves with Brian and Ash sharing their experiences of Manchester BarCamp. The arrangements had been better this year with half a dozen lecture rooms available. Brian had given his IoT talk which he had tried out on us the previous month and they had enjoyed sessions on Hacker Packet Radio and Git.
David S then managed to share with us an episode at work when a colleague had alerted him to a message via Sentry.
`InvalidSchema: No connection adapters were found for 'file:///etc/passwd'`
Because they use Apache with a reverse proxy, namely, uWSGI, David had to trawl through the uWSGI server’s logs, finding a wide range of attempts to crack the server leading up to the attempt which had prompted the warning at 10.40 am on 3 October.
A look in AbuseIPDB showed that the IP address from which these attempts were being made was in Ukraine though the actual attacked could have been in another country and it appeared they were using the acunetix web scanner.
As one of their customers had been involved in an anti-bribery initiative in Ukraine, one line of enquiry was that this might be an attempt at a revenge attack. However, analysis of the logs showed that servers related to other customers, in one particular data centr’’s netblock, had previously been gently probed by the same netblock in Ukraine, suggesting that the attack wasn’t targeted at the specific customer but just a coincidence.
This led into a discussion of how the organisation’s servers are managed and David said they used SALT partly because they can do everything using SSL. SALT uses ZeroMQ for messaging and YAML for scripting.
After this brief introduction, David was press-ganged into doing a more detailed presentation next month.
Brian gave a demonstration of live messaging between ‘things’ using MQTT in which members were encouraged to participate; this involved installing Mosquitto, a message ‘broker’ for MQTT, and then connecting to the temporary wi-fi network which Brian had set up.
Brian then moved on to demonstrating Node-RED, a Scratch like graphical interface for controlling the flow of messages between ‘things’ by software switching which can be installed on a Raspberry Pi. It supports a wide range of messaging formats including Asterisk, UDP and Twitter. Once configured, the interface can be switched off.
Brian warned about the need to deal with ‘switch bounce’ which occurs because, in for example a 5V circuit, Logic 0 is generally associated with less than 0.8V and Logic 1 with more than 2.4V making the voltages between 0.8V and 2.4V carry uncertain values. Ideally one should use a hardware latch to stop the effects of ‘bounce’ but a software latch could also be used.
In the course of this very successful demonstration, documented here, David S and Nigel managed to write a script to reconfigure the flows in the Node-RED page using Mosquitto as Brian had not yet worked out how to secure the messaging flow [but has done so since!].
Matt then presented ‘Tales of Linux micro-benchmarks.’ While benchmarks are intended to measure overall performance, micro-benchmarks are intended to measure the effects of very small code changes rather than overall performance. If you look on the Internet, you will find plenty of conflicting views about them.
So he gave some examples of working with them. Siege is supposed to measure the response of a server to increasing load. However, the code includes a line which means that, in effect, it is measuring itself at the same time as it is measuring the software.
Hackbench, a scheduler benchmark, needs to be run on a limited number of instances. In one case, Matt had found that 70% of the measurement related to setting up the system and only 30% to the software being measured because of the number of instances being measured.
Pipetest, which tests event delivery, produces some very pretty graphs but, when you look at the resolution of the graphs, it is so high that it is unhelpful. The resolution needs to be set to something useful.
So pick your micro-benchmark carefully and take care over the calls you are using. Make sure you understand what the data actually means.
Q&A; When might you use them? Mostly for parts of the system to which you do not have access or where you don’t want other parts of the system to influence the measurement.
Are they primarily to enable developers to identify regressions? That and also for capacity planning.
John H rounded off the evening with a quick review of the year.