Welcome to the Bradford GNU/Linux site; we are a self-help group for GNU/Linux, Open Source and Free Software users in and around the Bradford District, West Yorkshire
Why not come along to a meeting.
As only David had come with anything to share, we rambled round a wide range of topics.
Nick, who was with us for the first time since 2015, showed us the ThinkPad he had bought for £80 on eBay and told us that he had moved on from SkyBet to Leeds University Department of Engineering where there is a lot of Linux, mostly CentOS and using Puppet, and a wide range of computing resources up to an HPC cluster which is used by, among others, the European Space Agency.
He asked whether anyone had any thoughts on using
btrfs at which David went apoplectic while John said that SUSE Enterprise had been using it for some time apparently without problems but, when they had introduced it to openSUSE just over a year ago, a lot of people had found it filled up their root partition suggesting that it is best used by those who know how to use it.
David shared the paper by Vegard Nossum and Quentin Casasnovas on applying American Fuzzy Lop to filesystems given at Vault 2016. This showed that it had taken only five seconds to encounter a bug in
btrfs but two hours in
David then introduced the Repology website which tracks the versions of packages in the various repositories along with the maintainers.
He next updated us on how he had decided to automate kernel updates in Slackbuilds. He had settled on using
grep to search for the relevant words in the kernel.org RSS feed, save the results in
releases.new and then compare that with
releases.csv. If there are any changes,
releases.new is renamed
releases.csv and the building commences for the four kernels, 32bit and 64bit for the stable and the current (beta) versions. At the moment, this is all in a box running 24/7 because he has not yet set up the wake up LAN.
David then asked us about Patrick McHardy. The background is that the Software Freedom Conservancy and the Free Software Foundation published guidelines on GPL enforcement principles. There is GPL Violations, a German organisation founded in 2004, which sometimes pursues violators through the courts but Patrick McHardy has been pursuing violators in complete disregard for the principles and has had all his kernel commit rights taken away.
We then looked at the sketch on Youtube about Sean Spicer that had appeared on Saturday Night Live in the US.
Darren shared some of the problems which had appeared on the Open University TU100 My digital life forums relating to the SenseSense programming language which the Open University have developed from Scratch for use with mature students. Darren himself had had a problem because his 64-bit OS was just that; it had no 32-bit libraries.
Kriss and Shi Then shared their experiences of automating the development of static websites. Their first attempt was a ‘bootstrap’ CMS using App Engine and text files as input to generate the code which was tried out among the gaming community.
But they decided to step away from this in part because App Engine is aimed at large scale installations.
Their second attempt was ‘pancake’ using Open Resty which runs LuaJIT in Nginx to take advantage of Nginx’s efficiency. It was modular, used chunks and allowed multiple CSS files to enable different layouts within a website.
From this they moved on to ‘pated’ which has some similarities with Jekyll and decided to use the hat character as part of their naming convention; a directory starts with a hat character and file has one somewhere in its name. They illustrated that you can do quite a bit with static websites by demonstrating one showing aid data.
In practice, organising the data is the hardest part; the chunks are all in JSON format and you could use those on a dynamic website.
David C then gave us a running commentary on automating kernel security patches (of which there has recently been a rash) at Slackbuilds.
The idea is to have a single system that will notice patches to the stable kernel and apply them to Slackbuilds in such a way that builds will be reproducible, that is, however many people build it, all builds will be bit identical.
It needs something to read the RSS feed and apply
grep — in other words, a BASH script! This needs to prompt the router to wake up in order to start building the new kernel.
As he wants something simple, [If This Then That]](https://ifttt.com/) and Lighttpd on the router would seem to do the trick, though it was suggested he might want to consider Huginn instead of IFTTT.
This led into a discussion of the possibility of using a Twitter feed rather than RSS, in which case you would need to use https: with a single referer, and the availability of really cheap web servers.
Roger who hails from South Yorkshire and had stopped by on a return journey from Sutton Bank shared his experiences of using Linux with the PICkit and PIC microcontroller.
He had built his first computer in the 1970s, worked on a PDP-12 and programmed in Fortran, assembler, C+ and C++ mostly in the Steel Industry. Since he had retired he had been trying to show that you can do everything in Linux using GTK+ for which he had found formal descriptions but not always good examples. So he had decided to write his own examples.
As his hobby is gliding a lot of his examples relate to gliding. One is an anemometer head with optical readout which uses the PIC’s digital ports. He uses the Small Device C Compiler (SDCC) which is supported in PIC along with the GNU utilities.
You need to pay €20 for the PIC programmer because you cannot inspect the internals of a microchip or step through the C code line by line; so you need the simulator.
The PIC chip is a RISC chip and normally takes four cycles to execute a function; it also has analogue ports. Obviously, you have to understand the chip to decide what you want to do with it.
This led into a discussion of the merits of the PIC and Atmel chips.
Roger went on to say that he had mostly done the backend at work; now he was doing the frontend to prove the concept that it could also be done in Linux. He had completed three projects to illustrate this:
a device to send signals back to the winch during glider takeoff
the anemometer head
a battery charger which also calculated the capacity of the lead-acid batteries used in gliders — his most complicated project to date.
Stephane said that, since getting his amateur radio licence, he had been working at 20MHz HF and 70m DMR and built his own antenna which could pick up signals from Leeds in Harrogate. He has made contacts with Australia, Canada, the US, Germany and France.
A ham contact is called a QSO and hams like to log them; among the options are:
but most logging software
is Windows based
lacks lifespan maintenance
is cluttered and complex with screens filled with buttons.
He wanted to:
understand all the possibilities of a log
sharpen his coding skills as Java 8 has added functions and predicates to its object oriented framework
make it his own.
Among possible formats for a log are
Using xml data and xml formatting it is practical to build logs to a specific format and the ADIF 3.4 documentation is all available.
So he decided to use a client-server model which also helps in competitions when each contestant can have a client and to use xml validation in both directions.
He built it using JDK, Eclipse and the Postman plugin for Chrome.
The server loads validated data and stores it in memory; the client sends one or more QSOs embedded in an ADS object within validated xml to the server and can request QSOs from the server in the same format or query the server database.
In response to questions about the database, Stephane said the database consisted of QSOs stored according to an xml format from which it was possible to generate Java objects. He had tried to make it as small as possible using the built-in Java server. One could hold the data in an enterprise server.
Java takes a chunk of RAM and allocates space for the database which can be loaded and dumped. It can also be validated in Java.
The xml schema is contained in an xsd file and the xml config file will have details of the log books and the specific services being used.
The REST API offered post and query and he had created a third option of ‘get last n QSOs.’
The client page is simply an HTML page running on node.js with a script pointing to the Java code. He used the W3C Chocolate CSS stylesheet. The client retrieves data from the server every three seconds.
Q&A; In response to a question about the server interface, he demonstrated some of the available commands —
freq — and how it would identify false entries. It also reports a successful log.
Asked about how time was recorded, he said there were options to enter time on and time off and to enter the time automatically, each of which could be edited to take account of time lags between an event on the radio and the entry in the log.
In answer to a question about creating a GUI, he said that he wanted the CLI for auto-completion and many radio amateurs were used to DOS style interfaces. He was thinking about adding a query feature. The server has 1,800 lines of code, the client a few hundred.
Asked about using JSON, he said that JSON is good for simple data and there are lots of JSON implementations which people who prefer JSON can use but he prefers xml. It would certainly be possible to put the request in xml and the response in JSON.
In response to a question about someone pulling the plug, Stephane said he had not taken account of that but could add a feature to perform dumps at set intervals to preserve the data.
John H then explained by way of introduction that the laptop he was using was his travelling laptop on which he had installed openSUSE LEAP 42.1 but had found the video unstable; some times it would run fine for hours; other times he had to reboot several times before it became stable. However, the USB hub on his main computer had decided it no longer wanted to operate bidirectionally and so he had moved everything on to this computer.
Then he found that
btrfs had taken up all the space on his root partition; so he had installed LEAP 42.2 with an
Ext4 root partition and begun to play with the LXQt desktop, a joint venture between some of the LXDE developers and the razor-qt developers which was in effect a Qt equivalent to Xfce rather than an alternative to LXDE.
He had found this very stable with KDE applications which had crashed in Plasma running smoothly though DigiKam, the KDE photo-editing program did not like it.
Further details are available from John’s website.
David S did a presentation on configuration management or how to make sure that everything you need is set up as you want it to be whether on one or on a thousand devices.
SALT uses Python modules which are extensible along with ZeroMQ for messaging. Scripts are written in YAML with Jinja2 providing functions. ZeroMQ has a temporary buffer in which to hold messages if there is a break in communication.
SALT is less impressive at scale but it can use SSH instead of ZeroMQ where there is no agent on a device whereas Chef, for example, requires an agent on every device. However SSH is not fault tolerant.
A master device holds all the config files which determine how minions are to be configured. Note that minions cannot see other minions. The target specifies which minions are to be configured; the state specifies how they are to be configured and the configuration is undertaken by Python modules.
A grain describes the configuration of the particular device on which it resides whereas a pillar holds the configuration for a minion on the master. A file server on the master holds the files which a master will send to a minion.
Setting up is primarily a matter of exchanging keys. The YAML files primarily contain data structures but YAML is sensitive to indentation while Jinja2 handles commands.
SALT has its own command line options and can be run on Windows by using Chocolatey.
David noted that configuration management needs to be see as part of a wider picture which includes provisioning, the Cloud, containers, monitoring and workflow in which each program has its strengths and weaknesses.
He concluded with a summary of what SALT is not very good at, quite good at and very good at.
John H raised a problem of sub-domains on a website disappearing from time to time following a move to Cloudflare and the consensus was that the problem would be a badly configured
nginx config file.
As no-one had prepared anything specially for the meeting and David S was occupied trying to get Adobe Flash to work on John W’s computer, we chatted among ourselves with Brian and Ash sharing their experiences of Manchester BarCamp. The arrangements had been better this year with half a dozen lecture rooms available. Brian had given his IoT talk which he had tried out on us the previous month and they had enjoyed sessions on Hacker Packet Radio and Git.
David S then managed to share with us an episode at work when a colleague had alerted him to a message via Sentry.
`InvalidSchema: No connection adapters were found for 'file:///etc/passwd'`
Because they use Apache with a reverse proxy, namely, uWSGI, David had to trawl through the uWSGI server’s logs, finding a wide range of attempts to crack the server leading up to the attempt which had prompted the warning at 10.40 am on 3 October.
A look in AbuseIPDB showed that the IP address from which these attempts were being made was in Ukraine though the actual attacked could have been in another country and it appeared they were using the acunetix web scanner.
As one of their customers had been involved in an anti-bribery initiative in Ukraine, one line of enquiry was that this might be an attempt at a revenge attack. However, analysis of the logs showed that servers related to other customers, in one particular data centr’’s netblock, had previously been gently probed by the same netblock in Ukraine, suggesting that the attack wasn’t targeted at the specific customer but just a coincidence.
This led into a discussion of how the organisation’s servers are managed and David said they used SALT partly because they can do everything using SSL. SALT uses ZeroMQ for messaging and YAML for scripting.
After this brief introduction, David was press-ganged into doing a more detailed presentation next month.
Brian gave a demonstration of live messaging between ‘things’ using MQTT in which members were encouraged to participate; this involved installing Mosquitto, a message ‘broker’ for MQTT, and then connecting to the temporary wi-fi network which Brian had set up.
Brian then moved on to demonstrating Node-RED, a Scratch like graphical interface for controlling the flow of messages between ‘things’ by software switching which can be installed on a Raspberry Pi. It supports a wide range of messaging formats including Asterisk, UDP and Twitter. Once configured, the interface can be switched off.
Brian warned about the need to deal with ‘switch bounce’ which occurs because, in for example a 5V circuit, Logic 0 is generally associated with less than 0.8V and Logic 1 with more than 2.4V making the voltages between 0.8V and 2.4V carry uncertain values. Ideally one should use a hardware latch to stop the effects of ‘bounce’ but a software latch could also be used.
In the course of this very successful demonstration, documented here, David S and Nigel managed to write a script to reconfigure the flows in the Node-RED page using Mosquitto as Brian had not yet worked out how to secure the messaging flow [but has done so since!].
Matt then presented ‘Tales of Linux micro-benchmarks.’ While benchmarks are intended to measure overall performance, micro-benchmarks are intended to measure the effects of very small code changes rather than overall performance. If you look on the Internet, you will find plenty of conflicting views about them.
So he gave some examples of working with them. Siege is supposed to measure the response of a server to increasing load. However, the code includes a line which means that, in effect, it is measuring itself at the same time as it is measuring the software.
Hackbench, a scheduler benchmark, needs to be run on a limited number of instances. In one case, Matt had found that 70% of the measurement related to setting up the system and only 30% to the software being measured because of the number of instances being measured.
Pipetest, which tests event delivery, produces some very pretty graphs but, when you look at the resolution of the graphs, it is so high that it is unhelpful. The resolution needs to be set to something useful.
So pick your micro-benchmark carefully and take care over the calls you are using. Make sure you understand what the data actually means.
Q&A; When might you use them? Mostly for parts of the system to which you do not have access or where you don’t want other parts of the system to influence the measurement.
Are they primarily to enable developers to identify regressions? That and also for capacity planning.
John H rounded off the evening with a quick review of the year.
John H announced that David C was moving back to the area after his wife had obtained a job in Leeds.
John showed his Intel Compute Stick; unfortunately, we did not have a female HDMI connector to enable it to be demonstrated.
Darren described the on-going saga of trying to get Slackware 14.2 running with LVM where he had made progress but not found a complete solution.
Website: as we need to upgrade the website and move it to its own dedicated server, John said it was hoped to get together a group who could work on the upgrade and move. Dave C had indicated his interest and David S confessed to having had his arm twisted to contribute.
There was general agreement that we should move from Wordpress to a purely static website as the dynamic aspects of Wordpress have hardly ever been used.
One option would be to use Github which does not charge for public repositories.
|Alice demonstrated the Sky Betting and Gaming Engineering blog which is written in Jekyll and hosted on [Github](https://github.com/skybet/skybet.github.io||). Jekyll uses Markdown, among others, to create the HTML code; layouts are the equivalent of themes in Wordpress. You can import Wordpress sites with the following code and add Disqus for comments.|
Linux Presentation Day: we discussed the possibility of putting something on to coincide with Linux Presentation Day on Saturday, 22 October 2016 but decided that, on balance, we could struggle to find a suitable venue or generate enough interest in the event and that we would do better piggy-backing such an event onto another local event such as Wuthering Bytes.
David S then demonstrated the workflow of Slackbuilds dot org, the community repositories for Slackware, which uses shell scripts to automate the build process. You need a shell script, a README file and the metadata in order to build the binaries from source. David is a moderator for submissions to Slackbuilds.
Stephane then demonstrated his new website which is based on presentation slides animated via the CSS file by reveal.js. Like Jekyll, is has Markdown support. There is a master presentation which whose style sets the style for all the other slides and it is well documented.
[When Stephane illustrated the HTML showing how each slide is designated by an
id= attribute, John H commented that the line
<meta name="viewport" content="width=device-width, ... in the <head> element is recommended by Google to deal with problems in some smartphones but actually ruins the display in other smartphones. As he has yet to receive any reports of problems from not using it, he does not use it.]
Stephane then described his search for secure messaging. The most common messaging protocol is XMPP, as implemented in, for example, Jabber servers, but it is not secure by default. So he looked at security options testing with the Openfire server which has an excellent user interface this time with OTR client side to secure the messages but the setup is a bit complicated as you each have to ask a question that you know the other person knows the answer to in order to validate the secure connection. Also XMPP clients with different OTR libraries might not allow encryption for incompatibility reasons.
There is also OMEMO which is mostly used on mobile ’phones but he was not able to obtain enough information on its crypto-security. Another option for smartphones is ChatSecure which builds on OTRv3. A German company is involved in ChatSecure. He has sent in some patches and can compile it.
The problem is that not everything works on mobile but the combination of OTR and GPG appears to be the way to go.
Brian suggested he have a look at Ring which is only in beta but uses X.509 certificates for security.
John H shared a video he had made of a student presentation on prototyping in 1987; students had been divided into groups of four to research a topic and his group had decided to present their results by way of a series of sketches. At the time development mostly involved COBOL and programming only started after the requirements had been fully specified which normally meant that, by the time the program was delivered, things had moved on and the program no longer met the needs of the organisation. The proposed solution was prototyping of a model of the program to get user feedback before embarking on the programming or building the entire application by prototyping through a series of iterations in much the same way as free and open source software is now developed.
John shared a couple of the figures that had been in the original report, one showing how the cost of hardware — which placed a premium on efficient code — had fallen below the cost of people in the 1970s, meaning that organisations would have to take more care of their people in the future, and the other showing that over half of the bugs in applications related back to the requirements stage and required 82% of the effort to rectify them.
While there was plenty of discussion in academia and computing journals of the advantages of prototyping and various examples from the US, the only example they had found in the UK had been at Boots.
The project had taken place one year before the first open source project initiated by Intel and it was striking how prototyping had come to underpin the whole free and open source software movement with Greg Kroah-Hartman recently stressing the importance of making a series of small changes when programming. John had also been struck by the similarity between some of the discussions in the 1980s and current discussions around the role of DevOps.
David asked him about the conversion of the 1980s video and John said the conversion to MPEG2 had been done using Windows [specifically the Honestech VHS to DVD converter and the MPEG2 file converted to mp4 using Kdenlive which offers a set of standard conversion options which are passed to FFmpeg to do the actual conversion. For old videos he uses the PAL 4:3 option. The Windows software comes with options to adjust the RGB channels thereby compensating for any deterioration in the video though this required some practice to get right.
Brian then mentioned that IF This Then That, demonstrated by David C on Monday, 15 December 2014, had been upsetting users by insisting on them using a new API and password. He had found Huginn which has lots of documentation but no introduction; so we watched the Introduction to Huginn on Vimeo, noting that you can extend its features using Ruby. Alice commented that it did similar things to Apache NiFi but appeared to have more features.
Darren asked for some advice on booting LVM in Slackware and it was suggested, after some discussion, that he follow the Slackware LVM instructions.
David S then asked for some advice on replacing UPS batteries which, with a number of caveats about using the correct batteries, was considered feasible.
He had used the Raspberry Pi 2 because it does 52 Mb/sec whereas the Raspberry Pi 1 only does 36; however, Scott said that the Raspberry Pi 1 can be configured for higher speeds.
Stephane announced that he has lost his job following the takeover of the financial software company he used to work for and has set up a limited company; he welcomes advice and contacts regarding possible work. His background is as a Linux sysadmin but he plans to take various courses to broaden his skillset.
Kriss and Shi introduced the latest version of Swanky Paint, a pixel editor inspired by Deluxe Paint on the Amiga, which they have just released on Steam. Written in Lua and running OpenGLES2, it runs on everything including Native Client. Among its features are:
it can take on the characteristics of older machines,
it can accept bae files from Blender
png format because this allows the saving of metadata as
json files within the
it exports to true colour
you can get rid of the GUI to speed up processes.
You can create texture maps in it and export them to Blender; these consist of a texture and a colour which are combined in the output.
Lua is a very simple interpreted language; they use LuaJIT (Lua just in time) which is almost a compiler.
It can run on a Raspberry Pi 1.
Stephane then announced that he has passed his first level Amateur Radio licence which led into a discussion of how you can take out the carrier from an AM broadcast, transmit it and then reinsert the carrier at the other end.
The next stage is to get the licence to build his own transmitter. Stephane noted that a lot of the early work on Linux was done by radio hams. The important thing is to avoid interference in the equipment. He is particularly interested in software defined radio and the CI-V protocol.
Brian recommended Linux in the Ham Shack and the conversation went on the talk about pirate radio stations, designing antennae, bouncing signals off the ionosphere and wavelengths.
John H talked about the background to and the work of Claude Shannon, the centenary of whose birth fell on 30 April 2016. As he was growing up ideas like ‘emergence’ (Charles Broad, 1925), the idea that things can have properties which cannot be deduced from full knowledge of their constituents, ‘uncertainty’ (Walter Heisenberg, 1925), ‘homoeostasis’ (Walter Cannon, 1932), ‘ecosystem’ (Arthur Tansley, 1939), ‘feedback’ (Norbert Wiener, 1943) and ‘network’ (Warren McCullough and Warren Pitts, 1943) were being defined for the first time, even though several had been described before then.
However, his 1937 Master’s thesis in which he proposed the application of Boolean algebra to digital circuits was later to be described as ‘possibly the most important … master’s thesis of the century’ (Gardner, Howard (1987) The mind’s new science: a history of the cognitive revolution, Basic Books, p. 144 ISBN 0-465-04635-5).
His 1940 PhD thesis was on applying mathematics to Mendelian genetics after which he became a National Research Fellow at the Institute for Advanced Study in Princeton, where he was in contact with Hermann Weyl, John von Neumann, Albert Einstein and Kurt Gödel.
After the US entered the war, he worked as a cryptographer at Bell Labs and various papers relating to this work including his ideas about ‘redundancy’ and ‘information entropy’ were published in The mathematical theory of communication (Shannon, Claude and Weaver, Warren (1949) University of Illinois Press).
Fritjof Capra and Pier Luigi Luisi (2014) The systems view of life: a unifying vision Cambridge University Press p. 93 ISBN 978-1-107-01136-6 summarise his work as follows:
> > Shannon realized that, in order to develop an effective mathematical theory of information, communication signals must be treated independently of the meaning of the message. Thus, the term “information,” as used in information theory, has nothing to do with meaning. It is a measure of the order, or nonrandomness, of a signal; and the main concern of information theory is the problem of how to get a message, coded as a signal, through a noisy channel. > > > > To measure the order, and thus the information content, of a signal, Shannon borrowed the concept of entropy from thermodynamics, where it is deﬁned as a measure of disorder ... He used probability theory to express the accuracy of the transmission of a given amount of information under known conditions of noise and was able to derive a formula that shows how the capacity of a channel to carry signals depends on its bandwidth ’(i.e., its theoretical signal capacity) and its signal-to-noise ratio (the measure of interference). > > > > Shannon made the surprising discovery that, even in the presence of noise, signals can be transmitted effectively, and that the capacity of the channel can be increased signiﬁcantly by adopting various coding schemes. Thus information theory became an important theoretical framework for coding and data compression in communication theory and computer science. > >
John H mentioned apropos the discussion at an earlier meeting about indexes based on substrings within columns that his brother had suggested these could be created in SQL by creating a column containing substrings from other columns and then indexing that column and it was suggested he investigate composite primary keys [these, however, are indexes based on more than one column, not substrings of columns].
David S then made another attempt to demonstrate BASH on Ubuntu on Windows but was frustrated by the fact that, even though he had downloaded all the available downloads before the meeting, by the time he wanted to start the demonstration there were fresh Windows updates which were still not complete by the end of the meeting.
So, instead, he tried out a USB WiFi dongle he had acquired at a knockdown sale; first step to run
lsusb to find the Vendor ID and the Product ID which turned out to be 2357 and 0101 making it a TP-LINK Archer T4U. A search on https://wikidevi.com/wiki/TP-LINK_Archer_T4U revealed that there was an open source Linux driver available which could be downloaded with:
git clone https://github.com/abperiasamy/rtl8812AU_8821AU_linux.git
Following the README and, after installing
dkms, David was able to make the driver and demonstrate it working as WLAN1 alongside the existing WLAN0.