SECURE STORAGE: DISKASHUR PRO 2 www.linuxuser.co.uk THE ESSENTIAL MAGAZINE FOR THE GNUGENERATION ULTIMATE RESCUE & REPAIR KIT � Digital forensics � Data recovery � File system repair � Partitioning & cloning � Security analysis INTERVIEW Vivaldi The web browser for Linux power users IN-DEPTH GUIDE The future of programming The hot languages to learn PRACTICAL PI Build an AI assistant Python & SQLite Micro robots! PAGES OF GUIDES > MQTT: Master the IoT protocol > Security: Intercept HTTPS > Essential Linux: The Joy of Sed Pop!_OS Get into Arch Linux The distro for creators, developers and makers 4 Linux distributions for entering the world of Arch ALSO INSIDE » Dev report: Kernel 4.15 » Java: Spring Framework » Top web admin tips THE MAGAZINE FOR THE GNU GENERATION Future Publishing Limited Quay House, The Ambury, Bath BA1 1UA Editorial Editor Chris Thornett chris.thornett@futurenet.com 01202 442244 Designer Rosie Webber Production Editor Phil King Editor in Chief, Tech Graham Barlow Senior Art Editor Jo Gulliver Contributors Dan Aldred, Michael Bedford, Joey Bernard, Neil Bothwick, Christian Cawley, Nate Drake, John Gowers, Tam Hanna, 7RQL&DVWLOOR*LURQD0HO/ODJXQR3DXO2·%ULHQ-RQ0DVWHUV Katherine Marsh, Calvin Robinson, Mayank Sharma, Alexander Smith, Steve Wright All copyrights and trademarks are recognised and respected. Raspberry Pi is a trademark of the Raspberry Pi Foundation. Advertising Media packs are available on request Commercial Director Clare Dove clare.dove@futurenet.com Advertising Director Richard Hemmings richard.hemmings@futurenet.com 01225 687615 Account Director Andrew Tilbury andrew.tilbury@futurenet.com 01225 687144 Account Director Crispin Moller crispin.moller@futurenet.com 01225 687335 Welcome to issue 186 of Linux User & Developer In this issue » Ultimate Rescue & Repair Kit, p18 » The Future of Programming, p60 » Best Arch-based distros, p81 International Linux User & Developer is available for licensing. Contact the International department to discuss partnership opportunities International Licensing Director Matt Ellis matt.ellis@futurenet.com Print subscriptions & back issues Web www.myfavouritemagazines.co.uk Email contact@myfavouritemagazines.co.uk Tel 0344 848 2852 International +44 (0) 344 848 2852 Circulation Head of Newstrade Tim Mathers Production Head of Production US & UK Mark Constance Production Project Manager Clare Scott Advertising Production Manager Joanne Crosby Digital Editions Controller Jason Hudson Production Manager Nola Cokely Management Managing Director Aaron Asadi Editorial Director Paul Newman Art & Design Director Ross Andrews Head of Art & Design Rodney Dive Commercial Finance Director Dan Jotcham Printed by :\QGHKDP3HWHUERURXJK6WRUH\·V%DU5RDG Peterborough, Cambridgeshire, PE1 5YS Distributed by Marketforce, 5 Churchill Place, Canary Wharf, London, E14 5HU www.marketforce.co.uk Tel: 0203 787 9001 We are committed to only using magazine paper which is derived IURPUHVSRQVLEO\PDQDJHGFHUWLΐHGIRUHVWU\DQGFKORULQHIUHH manufacture. The paper in this magazine was sourced and produced from sustainable managed forests, conforming to strict environmental and socioeconomic standards. The manufacturing SDSHUPLOOKROGVIXOO)6&)RUHVW6WHZDUGVKLS&RXQFLOFHUWLΐFDWLRQ and accreditation Disclaimer All contents © 2017 Future Publishing Limited or published under licence. All rights reserved. No part of this magazine may be used, stored, transmitted or reproduced in any way without the prior written permission of the publisher. Future Publishing Limited (company number 2008885) is registered in England and Wales. Registered RIΐFH4XD\+RXVH7KH$PEXU\%DWK%$8$$OOLQIRUPDWLRQ contained in this publication is for information only and is, as far as we are aware, correct at the time of going to press. Future cannot accept any responsibility for errors or inaccuracies in such information. You are advised to contact manufacturers and retailers directly with regard to the price of products/services referred to in this publication. Apps and websites mentioned in this publication are not under our control. We are not responsible for their contents or any other changes or XSGDWHVWRWKHP7KLVPDJD]LQHLVIXOO\LQGHSHQGHQWDQGQRWDIΐOLDWHG in any way with the companies mentioned herein. If you submit material to us, you warrant that you own the material and/ or have the necessary rights/permissions to supply the material and you automatically grant Future and its licensees a licence to publish your submission in whole or in part in any/all issues and/or editions of publications, in any format published worldwide and on associated websites, social media channels and associated products. Any material you submit is sent at your own risk and, although every care is taken, neither Future nor its employees, agents, subcontractors or licensees shall be liable for loss or damage. We assume all unsolicited material is for publication unless otherwise stated, and reserve the right to edit, amend, adapt all submissions. ,661 Welcome to the UK and North America?s favourite Linux and FOSS magazine. I?m writing this on the day that it was discovered that anyone can ?hack? Apple?s macOS High Sierra by clicking a prompt and typing ?root? in the username ?eld. I laughed, but I shouldn?t as it?s quite a ridiculous security failure that should never happen. Back in the world of Linux, we?re looking at ways to reduce the chances of failures of a different kind with our Ultimate Rescue & Repair feature (see p18). We explore what to do when disaster strikes: how to analyse issues, recover from them, and clean and maintain systems to avoid problems in the future. For our second feature, we consider what programming languages we?ll be using in ten years? time (p60). As in the world of programming, the past offers clues to what technologies will be big in the future and that?s re?ected in a new three-part tutorial series, where we cover MQTT. This 18-year-old protocol is extremely hot now because of the growth in IoT. We?re also coming to the end of our Learn Java series (p54), so email us with what languages you?d like to see in the magazine. Thanks for joining us this year as it draws to close. We hope you?ve enjoyed the magazine as much as we have enjoyed making it for you! Chris Thornett, Editor Getintouchwiththeteam: linuxuser@futurenet.com Facebook: Twitter: facebook.com/LinuxUserUK @linuxusermag filesilohelp@futurenet.com For the best subscription deal head to: myfavouritemagazines.co.uk/sublud Save up to 20% on print subs! See page 30 for details www.linuxuser.co.uk 3 Contents 18 50 ULTIMATE RESCUE & 60 THE FUTURE OF PROGRAMMING LANGUAGES REPAIR KIT OpenSource Features Tutorials 06 News 18 Ultimate Rescue & Repair Kit 32 Essential Linux Intel and AMD develop an revolutionary new CPU for 2018 10 Letters Scribbled musings from readers 12 Interview Vivaldi CEO, Jon von Tetzchner discusses privacy and net neutrality 16 Kernel Column The latest news on the Linux kernel 4 While we all know that Linux is one of the most stable and reliable operating systems around, things can occasionally go wrong. However, there?s is no need to despair if they do ? Neil Bothwick reveals the best tools for diagnosing and ?xing system problems 60 Future of Programming Languages Looking into his crystal ball, and extensive knowledge base, Mike Bedford examines the future of programming languages. Starting with unusual ones that broke the mould, he moves on to up-and-coming languages before getting the views of experts about what the future holds Explore the Joy of Sed 36 Essential admin commands 20terminalcommandsyouneedtoknow 40 Video capture RecordwithSimpleScreenRecorder 42 MQTT AnintrototheprotocolpoweringIoT 46 Security RedirectHTTP(S)requestsfromdevices 50 Arduino Usesleepmodetomonitortemperature 54 Java Spring Framework: dependency injection Issue 186 December 2017 facebook.com/LinuxUserUK Twitter: @linuxusermag 94 Free downloads We?ve uploaded a host of new free and open source software this month 86 72 74 76 8 Practical Pi Reviews Back page 72 Pi Project: Micro robots! 81 Group test 96 Near-future fiction Seeking a better way to teach multi-robot behaviour in class has led Joshua Elsdon to create micro robots that are smaller than the change in your pocket ISSUE 186 86 Hardware & DEVELOPER THEESSENTIALMAGAZINE FOR THE GNUGENERATION The iStorage diskAshur Pro2 external ULTIMATE hard drive offers solid security, but it RESC REPAI is worth the premium price tag? 78 Pythonista?s Razor Discover how to use SQLite with Python to create a database to manage all your precious information 90 Fresh FOSS CudaText 1.23.0 code editor, Subsurface 4.7.1 dive log tool, LXQt 0.12.0 DE and Converseen 0.9.6.2 image converter MAS Vivaldi The web brow Linux power u 88 Distro spotlight Linux hardware vendor System76 has put together its own distro, based on Ubuntu, called Pop!_OS IDES INTERVIEW PRACTICAL PI Build an AI as Python & SQL Micro robots www.linuxuser.co.u Add voice-activated artificial intelligence to your Raspberry Pi with the aid of Google Assistant and ReSpeaker pHAT GES OF hell ipting aptop wer tips uild a t sentry ics � Data reco � Digital forensoning & cloning repair � Partiti & REPAIR KIT 76 AI assistant uk MAGAZ NE THE ESSENTIALENERATION FORTHE GNUG ULTIMATE RESCUE Learn how to draw vector shapes directly into the Minecraft world using just a few lines of Python code www.linuxuser.co.uk www linuxuserco LINUX USER 74 Minecraft & Python Experience the bene?ts of Arch in the Chained to a super-smart mirror, there comforts of a desktop distribution. is a strong desire to break free Which one of the four solutions will FREE DVD UBUNTU 17.10 SP CIA NS 2 UBUNTU SPI come out on top? FREE DVD TRY Pop! OS creators The distro for makers deve opers and Linux Get into Archs for 4 L nux d stribution d of Arch entering the wor L P o2 » Storage D skAshur Framework » Java Spr ng Wi Fi » Disaster re ief priva SUB » pen sou ce microscopes BE TODAY Save %when you subscribe! Turn to page 30 for more information www.linuxuser.co.uk 5 06 News & Opinion | 10 Letters | 12 Interview | 16 Kernel Column HARDWARE Intel & AMD develop revolutionary CPU Surprise joint venture yields results: new chip for 2018 release offers high-performance, discrete graphics What would you say to a powerful new CPU from Intel with built-in graphics from AMD? Nice idea?and something you?re going to be able to buy soon. Rumours of an Intel-AMD collaboration have been swirling since early 2017, but it seemed just too farfetched. After all, the former partners have a history of litigation between them. Even more surprising is that the venture has yielded results so swiftly. Clearly noting the obvious weaknesses in its own graphics processors, Intel has done the sensible thing and brought on board the only company that could appreciate the engineering dif?culties of balancing a CPU and discrete GPU. But not only does this cessation of a 30year rivalry deliver a new product; it could also revolutionise computing over the coming years, offering OEMs the freedom to develop lightweight, thinner designs. It doesn?t end there: improved thermal dissipation, new cooling solutions and increased battery life are all possibilities. ?Our collaboration with Intel expands the installed base for AMD Radeon GPUs and brings to market a differentiated solution for high-performance graphics,? said Scott Herkelman, vice president and general manager of AMD Radeon Technologies Group. ?Together, we are offering gamers and content creators the opportunity to have a thinner and lighter PC capable of delivering discrete performance-tier 6 graphics experiences in AAA games and content creation applications. This new semi-custom GPU puts the performance and capabilities of Radeon graphics into the hands of an expanded set of enthusiasts who want the best visual experience possible.? Introduced as part of the 8th Gen Intel Core series, the project uni?es Intel?s highperforming Intel Core H-series processor, its second generation GDDR5 High Bandwidth Memory (HBM2) and a custom discrete graphics chip from AMD?s Radeon Technologies Group. Intel?s Embedded MultiDie Interconnect Bridge (EMIB) technology is at the heart of this, supporting the processors with a revised power-sharing framework. Basically, EMIB is a bridge that speeds up the ?ow of data, enabling faster, more powerful and ef?cient devices. Above Intel Inside or should it be AMD? If it?s thin and lightweight portable it can now be both! all obvious homes for this new technology. Meanwhile, notebooks, hybrids, all-in-ones and mini desktops are all set to become lighter and more powerful. As for gaming, thinner devices, integrated gaming TVs and lighter games tablets are all possibilities. The implication for homebased media production, meanwhile, is considerable. But some things are less clear. So far, we don?t know how much the new processor will cost. Similarly, neither Intel nor AMD has advised whether the processor will be available beyond the OEM market. Oh, and it doesn?t even have a proper name yet. Intel Core Single Package Multi-Chip Solution doesn?t exactly roll off the tongue, does it? Intel has brought on board the only company that could appreciate the engineering dif?culties of balancing a CPU and discrete GPU But what does all this mean to GNU/Linux and the open-source world? At this stage it is too early to say, but thinner devices aren?t just bene?ts for gaming. Mobile hardware, media centres and smart home devices are DISTRO FEED Top 10 (Average hits per day, month to 17 November 2017) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. WEB DEVELOPMENT Mint Ubuntu Debian Manjaro Antergos Solus elementary Fedora openSUSE TrueOS 2032 1825 1616 1323 1321 1090 960 930 857 This month Firefox 57 ?Quantum? challenges Chrome QStable releases (5) QIn development (4) Another Debian-heavy month in the top 10. Meanwhile, several distros are gaining interest for their Windows-like desktop environments? Faster and stronger browser hits Ubuntu Billed as the biggest update to hit Mozilla Firefox in the browser?s 13 years, Firefox 57?dubbed ?Firefox Quantum??is here. But does it make the quantum leap needed to combat the popular browser?s decline? Featuring 4,888,199 new lines of code and taking a year to produce, Firefox Quantum has taken an important step into the future. This version of the browser ?nally casts off the shackles of the legacy extensions, providing compatibility only with web extensions. While this means a good number of older add-ons will no longer work, 6,000 working web extensions are listed. A muchneeded stability and security improvement, the bene?ts outweigh the shortcomings. ?We couldn?t do any of this without our loyal and rabid users,? blogged the Firefox Quantum team, ?Who make us try harder, work later, code longer, cheer louder and ?ght for more and more of what?s right in keeping the internet open, safe and exciting.? So can Firefox ?ght back against Chrome? Although it will take a while for the ?gures to re?ect it, the early signs are good. Most of the excitement is focused on the 2766 performance improvements, which are widely considered to be twice as fast as 2016?s Firefox 49. Notably, Quantum uses 30 per cent less memory than Google Chrome, partly due to changes under the hood, and also thanks to the new CSS engine, Stylo, and a new multi-process architecture. Public reaction on Twitter and Reddit has been positive, and there?s a buzz around Firefox that hasn?t been seen since the noughties. Firefox Quantum offers a number of features to enhance the browsing experience. Rather than system screenshots, Firefox enables you to snap them in the browser. Meanwhile, the Pocket read-later-syncing service is built into the browser. If you have Firefox installed on other devices, you can keep reading. Pocket saves, screenshots and even tabs can be found in Firefox?s own cloud library. The browser also supports WASM and WebVR for next-generation gaming, new themes and toolbar recon?guration. Firefox Quantum can be manually downloaded from www.mozilla.org, although Ubuntu 14.04 and later users will receive the update automatically. Highlights PCLinuxOS Perhaps the most famous of the Windowlikes, PCLinuxOS comes with out-of-thebox support for many graphic and sound cards, and runs most popular hardware. The grey toolbar and menu deliver a strong Windows feel, while a user forum exists for troubleshooting. Zorin OS Based in Ireland,Zorin OS is perhaps the ultimate Windows-esque Linux distro. As well as offering a start menu experience similar to Windows 10,Zorin OS ships with Wine pre-installed. antiX Fast and lightweight, antiX uses a classic Windows-style desktop taskbar and start menu reminiscent of Windows XP. Based on Debian, antiX can also run on old Pentium III systems. Latest distros available: ?lesilo.co.uk www.linuxuser.co.uk 7 OpenSource Your source of Linux news & views DISTRO Slax abandons Slackware for Debian New release retains Slax name, not yet being renamed ?Dex? Slax 9 is now available for download, but in a surprise move developer Tomas Matejicek has announced that the new version of the lightweight distro has abandoned its roots. So if Slax isn?t based on Slackware, what?s running underneath the leafy new desktop background? Well, it?s Debian. Outlining his reasons with clarity, Matejicek has basically stated that it?s too complicated to maintain Slax if it continues to be based on Slackware. ?The reason is simple: laziness,? Matejicek blogged. ?I am too lazy, really, really lazy. When I prepared Slax from Slackware, I had to patch kernel with aufs, con?gure, recompile, and so on. Then compile other software from sources, ?ght dependencies, and so on. I enjoyed doing that in the past, but now I?m not in the mood anymore.? To prove Matejicek?s key point (?all Linux distros are the same anyway, ... it?s all Linux?), the new version of Slax What?s running underneath the leafy new desktop background? Well, it?s Debian Above Slax 9 looks great, but has abandoning Slackware sacri?ced users? is just 208MB (or 218MB for the 32-bit release). Fitting comfortably on all but the smallest USB sticks, there?s probably just one key argument against Slax becoming (unof?cially) Dex: the arrival of systemd. Meanwhile, the desktop arrives with just four icons: the Terminal, Text Editor, Calculator, and Chromium Browser. Matejicek?s rationale is simple: he believes everything is moving to the web, so why spend time packaging tools into an OS? Slackware has remained systemd-free, with the bootstrap and process manager often cited as a tipping point for Slackware adherents. It seems likely the move to Debian (and implicit acceptance of systemd) will lose some users, but Matejicek seems con?dent that it doesn?t really matter. PUBLIC SECTOR Pentagon switching to open source in 2018? DoD says updates to closed-source software are too slow Could the Pentagon be abandoning proprietary software foro open source? That?s the implication of a key portion of the National Defense Authorization Act for Fiscal Year 2018, in an amendment introduced by Sen. Mike Rounds and cosponsored by Sen. Elizabeth Warren. But why? First, as the world?s biggest single employer, the Department of Defence has a massive IT requirement. A lot of money is spent on data warehousing, statistics, brie?ngs, presentations, spreadsheets and documents. Licencing costs are expensive. 8 Second, there is a perception that waiting for closed-source applications to update puts the DoD at a disadvantage. Rather than playing catch-up, it?s preferable to adopt new software tools. Procurement takes time, both in terms of signing off requests and waiting for a release date. Then there?s the roll-out, another potential delay. Of course, this amendment is not without its detractors. The security argument is being highlighted. For instance, Brian Darling, president of Liberty Government Affairs, has written in the conservative publication Townhall challenging the proposal, citing national security concerns. Conversely, Federal Chief Information Of?cer Tony Scott called in 2016 for government-speci?c opensource code to be built, demanding it be ?secure, reliable and effective.? Supporters of closed-source solutions seem to have overlooked that international rivals have personnel capable of spotting and exploiting ?aws in proprietary software. At this stage, and in the wake of Munich ditching Linux, it seems likely the amendment will be defeated. TOP FIVE Open source web browsers 1 Mozilla Firefox Quantum HARDWARE Critical bugs found in Intel Management Engine Security vulnerabilities also identi?ed in SPS and TXE technologies Using an Intel PC or laptop? Your computer is at risk. Hidden away in Intel?s Management Engine (ME)?the secret ?computer within a computer? whose existence was only con?rmed in 2016? are a collection of bugs that could enable any Intel-equipped Linux, Apple Mac and Windows computer to be hijacked, even with no operating system installed. Third party security experts (UK-based Positive Technologies) have discovered that along with ME, Server Platform Services (SPS) and Trusted Execution Engine (TXE) are susceptible to exploitation by anyone posing as a network administrator. The attacker may then install spyware and rootkits. Once a user is logged in, malware or hijacked applications can be subverted to leak data from the system RAM. Shared machines and servers are particularly at risk, and code to exploit ME can be introduced via USB stick. As Intel?s Management Engine was ?rst introduced in 2008, the overwhelming majority of processors currently in use have the vulnerability. This includes 6th, 7th, and 8th generation Intel Core CPUs, lowend Celerons and Intel Xeon server chips. Described as a ?backdoor? by some security commentators, ME is essentially a mini computer with its own CPU and operating system (since 2015, the UNIX-like MINIX). Perhaps most worryingly, ME runs beyond your OS, out of the reach of antivirus tools. So why is it included? Intel?s Management Engine is ostensibly provided to allow network admins to remotely access servers and workstations, ?x errors, provide desktop support and reinstall the OS from afar, which are all vital tools in many organisations. Positive Technologies? researcher Maxim Goryachy explains: ?An attacker could use this privileged access to evade detection by traditional protection tools, such as antivirus software. Our close partnership with Intel was aimed at responsible disclosure, as part of which Intel has taken preventive measures, such as creating a detection tool to identify affected systems.? Intel?s detection tool (INTEL-SA-00086) can be grabbed from its download center. Meanwhile, patches have been rolled out. However, don?t expect any patches from your operating system. If your system is affected, you?ll need to contact your hardware?s OEM. Released amid a storm of publicity, Firefox 57 (known as Quantum) has proven popular. The biggest update in Firefox?s history, Quantum is faster than Google Chrome, and uses 30 per cent less memory. These changes may secure Firefox?s future, but are they enough to challenge Chrome? 2 Midori Employing the WebKit rendering engine, Midori is associated with the Xfce desktop and developed to the same principle (?making the most of available resources?). With a low footprint, it can be found on many light Linux distros. 3 Chromium Anyone who has used Google Chrome should be comfortable with its open source sibling. Much of the same functionality is there, along with support for extensions. Chromium is fast and easy to use. 4 Brave Based on Chromium and spearheaded by Mozilla Project co-founder Brendan Eich, Brave shares less data with advertisers and blocking trackers. 5 GNOME Web Formerly known as Epiphany, this is still going strong (the package retains the name epiphanybrowser), and utilises the WebKit rendering engine. Recently, some integration with Mozilla Firefox Sync has been added, enabling continuity across devices. www.linuxuser.co.uk 9 OpenSource Your source of Linux news & views COMMENT Your letters Questions and opinions about the mag, Linux and open source Above What events would you like to see covered next year? Email us at the address below GET IN TOUCH! Got something to tell us or a burning question you need answered? Email us on linuxuser@ futurenet.com 10 Key conferences Dear LU&D, I started reading your Linux magazine about a year ago. It?s a great read, and I have learnt a vast amount from the tutorials and the more practical features that you publish. I have also enjoyed your pieces on general Linux goings-on, stuff like Linux in Space. I have to say that one thing that has surprised me is the lack of an events page. I could do with something that tells me what training events and conferences are worth my time. I think that would be helpful to busy readers like myself who only want the key diary dates. Joshua Allen-Phelps Chris: Glad you?re getting so much out of the magazine, Joshua. Thanks for the topic suggestions that you supplied in my reply as well, it?s really helpful to get an idea of what people want to see in the magazine. Diary dates would be a good idea, but I?d be interested to see whether more people want this and what they?d like to see. Should we supply a Linux User Group (LUD) list, stick to professional events or do both? My concern would be to make sure we only cover the crucial events, as your time out of the of?ce is valuable. We?d also love to hear from organisers and attendees, as we?re always on the lookout for people who can report on the events for us. Email as at the usual place: linuxuser@futurenet.com. Right Making sense of the output of useful commands such as dmesg, fsck and lsmod will assist those trying to help you Crucial commands I?ve got a suggestion for a subject to cover in your excellent Linux magazine. I ?nd that when things go wrong and I head for the usual places to seek advice from more experienced heads that a post without some useful output can either get dismissed out of hand, or my post is swiftly followed by a request that I run various sysadmin commands. Knowing about the best ways to read the outputs of these commands is an important skill as well. Also, the ways that you can use commands to decipher the output would be appreciated too. There are so many options to choose from in the main pages, it?s sometimes hard to know what?s best. I know you?ve covered grep in the past, but there are other methods. Derek Harris FOLLOW US Chris: Thanks, Derek. Some of those commands are likely covered in our Ultimate Rescue and Repair feature on p18 of this issue, but what a great idea! It?s certainly something we can look at doing, especially as John Gower?s little series on mastering shell scripting is drawing to a close. I?m sure forum veterans would appreciate posts for assistance that demonstrate some thought into what information might help in getting to the root of the problem. Watch this space and we?ll see what we can do. Facebook: Twitter: facebook.com/LinuxUserUK @linuxusermag portion of our name needs more love. We?ve tended to focus almost entirely on the desktop OS story, but that needs to change, as the budding sysadmins and pros also need to know what features are going to bene?t their work-life. You?re right that there?s a lot of exciting things happening in the desktop universe?that?s why we have Antergos on the disc?but we always want to hear what distros have caught our readers? attention. We?ll likely cover Solus soon, but we also ran a poll to see if readers wanted Fedora 27. It gained 65 per cent approval from our Twitter followers (out of 285 votes). Not exactly a ringing endorsement, but enough for us to put it on the disc. Pixel OS in Dear LU&D, I?m interested in the Raspberry Pi, but I?m mostly keen on trying out the Raspbian OS. Can I install that on one of my old laptops as well, or do I need to buy one of the RaspPi boards? Philip Booker Chris: Yes you can, Philip. In December of last year, the Raspberry Pi Foundation released Pixel OS, an x86 version of Rasbian that uses the PIXEL desktop environment (which is modi?ed from LXDE). The OS is particularly suited to ancient machines and if your interest is based on coding, you?ll ?nd many environments bundled with it (including Python and Java) as well as environments for Pi add-ons such as the Sense HAT emulator. You can download it here: https://www. raspberrypi.org/downloads/raspberry-pi-desktop. TOP TWEET @andysc: ?Yes, it?s been a long journey, but very pleasing to see the ?dream? come true :)? Andy StanfordClark, the coauthor of MQTT, answering what it feels like to see the protocol taking ?ight 18 years later in today?s world of IoT devices. Follow us on Twitter @LinuxUserMag Below Pixel OS will have its ?rst anniversary as an x86 desktop OS in December Above Fedora 27 won a recent poll to be included on the LU&D disc, but what distros would you like to see in the future? Ubuntu out Dear LU&D, isn?t it time to drop all the Ubuntu coverage and launch packs? There?s so much more going on in the desktop world beyond Canonical, especially after it has dumped the desktop in favour of lots of dough from cloud and IoT customers. Larry Smith Chris: I wouldn?t say Ubuntu has ?dumped the desktop?; it still has a 17-person team working on the desktop version of Ubuntu. Admittedly, most of those are GNOME developers, but that?s still a sizable team. In some ways, Ubuntu?s focus makes it more relevant to LU&D. From the dribble of feedback we?re beginning to receive, it?s clear that the ?Developer? www.linuxuser.co.uk 11 OpenSource Your source of Linux news & views INTERVIEW VIVALDI Vivaldi: The power user?s browser We interview the CEO of Vivaldi, a Chromium-based browser, about the ?ght for privacy and net neutrality and what features are in the pipeline Jon Stephenson von Tetzchner has been blazing a trail in the internet-browsing business since 1995 as co-founder of Opera, and now as the CEO of Vivaldi. Below Vivaldi is striking out in the opposite direction to the main web browsers by offering multiple ways to do things 12 The Vivaldi browser started life in 2015 as both a response to a change of direction at Opera and what Jon Stephenson von Tetzchner, the CEO and past co-founder of Opera, sees as a demand for a web browser that caters for internet power users. Von Tetzchner believes that the main browsers are over-simplifying their experience: ?They are basically adapting to someone that doesn?t exist,? says Jon talking from Magnolia Innovation House outside Boston. ?Any feature that?s used by not so many people gets removed and the software gets dumber and dumber in a way.? Vivaldi is essentially walking in the opposite direction with signature features such as tab grouping, a built-in screenshot capture tool and the ability to take notes right in the browser that are heaped together with lots of customisation options. And while many browsers rely on add-ons, Vivaldi is releasing differentiating features that are built natively into the browser. Can you tell us about your overarching philosophy for the browser? Our feeling is that if you?re spending a lot of time on the internet, you actually need a browser that can adapt to your requirements and your needs. There should be multiple ways to do things. In some ways, in the way that you do things on Linux, you have a lot of ?exibility, a lot of different ways to do things, so you ?nd your way to do things and that?s what we?re all about. It?s about adapting to your requirements as a user by putting in a lot of features, but by also putting in a lot of ways to do the same thing so you can have it adapt to your requirements instead of the other way around. Have you experienced a notable interest from Linux users? Yes. If you look at the amount of Linux users we have, they match the number of Mac users, which tells you that Linux users are obviously ?ocking to us in much greater numbers than users of other platforms, which is nice. Are you around the 1 million users mark? It?s around that ?gure, we are saying that we?re closing in on 1 million. That?s kind of where we are. We?re getting pretty close to that. Our growth rate is increasing, we?re close to our ?rst million and we?re counting active users; we?re not counting downloads [?] or numbers that make no sense. We?re counting what means something to us, which is the number of active, monthly users. The Chrome browser is dominant on the Linux platform, even Mozilla has acknowledged that, and the amount of data being tracked now is signi?cant. Do you see Vivaldi as a solution? We will do what we can on our side, but actually, after spending a lot of time thinking about tracking and targeting, my feeling is actually this is a problem that needs to be solved in a different layer. That?s something I?ve decided to start talking about. I was getting the questions, ?What are you doing to help keep us safe?? and we started discussing whether we should put in Tor or something like that. There are a number of things we can do, and we?re looking into that. The trouble is that the moment you start using something like Tor, you actually look like you have something to hide. And that?s how the government views it, and I think that?s unfortunate that we, when we want to just be private, we?re seen as wanting to do something wrong. My feeling is that this needs to change, and there is a problem with the amount What?s new and coming? What?s impressive about Vivaldi is the features that are being released with each update. For instance, the last release, 1.13, focused on the Window Panel. This is a classic example of Vivaldi catering for power users who tend to have lots of tabs open. The panel presents a ?tree-style view of tabs to the side of the browser window for easy management. You can do such things as grab and drag tabs; group them by topic; use Tile Tab Stack feature to compare pages side by side, mute individual tabs or hibernate tabs for better performance: ?It?s another way to do tabs,? says von Tetzchner, ?and we just know that people have very different opinions and strong opinions how they should do their tabs.? The Vivaldi team has worked on improving the Downloads panel as well, as von Tetzchner says: ?We view downloads differently from the other browsers. We have the download panel [...], which has a lot more information.? This time, Vivaldi has included the ability to pause and resume downloads; it?s also added supply speed information in the progress bar, and a warning dialog when closing the browser mid-download, which are all important for Vivaldi users according to Jon, as they download more than the average user. He also says that while Vivaldi doesn?t have BitTorrent currently, it?s one of the items on the development list, as it was included in Opera. In terms of the future, while von Tetzchner says a lot of the focus is on ?nishing off existing tasks, such as the email client and mobile, he?s got his eye on peer-to-peer. ?From a privacy perspective, I think that?s great. I don?t think everything has to go through the of information that is being collected, as it is too great and the amount of targeting is too great. We?re somewhere on the way, just because the technology provided us with the way to do this, the industry said that?s okay to do, and I think it?s wrong. The problem needs to be ?xed at a different level, basically. You have talked about regulation in the past. What form of regulation do you feel is required for online search and advertising? There are two things, and in some ways it?s not that big a change. It?s basically going back to where we were a few years ago, and I?m not the only one who has been talking out about this?Sir Tim BernersLee has also made comments in this direction. It?s something that has been seen by a number of us who have been on the internet for a long time that something is going in the wrong direction here. It?s rather simple: you shouldn?t be allowed to collect more information than you need, and you shouldn?t be able to use the data for things that are not relevant to the task that?s being performed. As an example of that, there is no need to collect information about your location at any one time, and there?s no need to provide that to third parties either. I mean, the location information is needed when you as a user want the location information. That?s when that information is useful. So you want to use it for maps, you want to use it for driving in traf?c, or maybe want to have it on for a while because it helps you. There are two things: how much are you Above Vivaldi is constantly adding features. Recently that?s meant a new Window Panel for management tabs (pictured), but synchronisation is due late November cloud [...]. Too much is going through the cloud, which is a security and privacy issue,? says Jon. In the short-term, Vivaldi is due to release its most most-requested feature: synchronisation on 29 November, and in early December it will release an experimental build for the Raspberry Pi and ARM devices. collecting and keeping it, and how much are you selling it? Or using it for other purposes that were intended? There are a lot of things I can say I don?t mind?for example, the concept of recognising voices is really nice, right? And being able to translate, the Google stuff that?s shown is fantastic, but the question is, can we trust Google not to use that information for anything else? Does it need to go into the cloud at all for the translation purposes? This is the question and what we?ve been seeing, at least with the collection of data and how it?s being used; it?s being used in forms that have gone too far. It used to be that advertisements were based on the location. You?d know something about the audience that was in that location and you would be able to use IP to decide where in the world they are from, but you weren?t able to do localised advertisements based on the person. You weren?t able to follow a person from page to page. That praxis is very unfortunate, and needs to be changed. You have stated in the past that advertising data should be freely available. How do you envisage that working? If you see an ad, whether it?s coming through Facebook or Google [?] you should be able to see who provides that ad. You should also be able to see all the ads provided by that same party to get an impression of what this person is doing. I don?t think that?s enough to deal with how ads are being used by some. I mean, supposedly Trump sent It?s unfortunate that we, when we want to just be private, we?re seen as wanting to do something wrong www.linuxuser.co.uk 13 OpenSource Your source of Linux news & views The licensing elephant You can?t cover Vivaldi without addressing the elephant in the room, and that?s the browser?s current lack of a full free or open source licence. It is a complicated situation, as all the changes made to the Chromium engine source code (available at https:// vivaldi.com/source) are released under a BSD licence. This type of licensing doesn?t require source code to be distributed, although Vivaldi does, and also makes the UI code accessible to read in plain text. ?Every change we do to the C++ code? that?s open,? says von Tetzchner. ?When it comes to the HTML side, we haven?t decided on a licence. We have had discussions? should we put it out with a licence? But we?ve not really done that.? And people are able to go in and change the code as long as they know HTML, JavaScript and CSS, but Vivaldi isn?t actively encouraging it: ?People can?t then take that and build their own [browser] on that, but at the very least people are able to go check the code and have a closer look at what we?re doing?that we do like.? Von Tetzchner says that the company did consider using GPL, but that it?s a dif?cult decision because of the risk involved: ?By picking the wrong licence and putting it out there, we?re really concerned by that, and we do feel that, in many ways, people can still do what they like,? says Jon. It also hasn?t deterred the community from contributing code, as some users have put out patches and sometimes submitted patches for incorporation. Sometimes these patches have been included in the browser, but von Tetchner acknowledges that mostly they don?t make it in. It seems that Vivaldi?s biggest fear is the distribution and sharing requirements of some free and open source licenses: ?That?s probably our biggest risk, yes,? says von Tetzchner. ?Our impression is that people like what we?re doing [...] I think the way we do things as a company, we?re more open than anyone else [...]. It?s all very open with how we work with people. The source is not something that comes up on a regular basis, it?s mostly when I talk to Linux magazines, which is reasonable, and it is a dif?cult thing, as a lot of us like Linux and a lot of us like open source, so it?s complicated.? out 40,000 different ads per day [During the US Presidential Elections] and clearly going through that is not feasible, so I would say that it needs a programmable interface to that where you could use big data to actually analyse the ads that are being sent out [?] so we can analyse if someone is doing the kinds of ads you don?t want to be seeing; that people are sending out different ads to different people with a different message to get them to act in a certain way. I mean, from a democratic perspective that?s important for us to know when people are doing that. Our online services have not been regulated in the same way as maybe television and other channels have been, and we are seeing this is an even more potent channel that needs to be, if anything, more closely scrutinised. We?ve seen some companies given tools to do bad things will do bad things. This is a very powerful tool to do bad things if you want to. How does Vivaldi make its money if it?s not tracking user data? Most of the revenue from browsers is from search, so we have a few search deals, and then we have a few bookmarks that we include that generate revenue [?]. The amount of revenue we?re calculating 14 LGPL BSD GPL APACHE MIT CC Above Vivaldi?s CEO says that the company faces a ?dif?cult decision? when it comes to what uni?ed open source licence to use for its innovative browser with is about a dollar per user per year. We can do things without spying on our users, and I think it?s a lot more fun building great products for our users and being on their side, and okay, if we can make a dollar per user per year, that?s enough to pay our bills. Net neutrality is in serious danger in the US. Do you think there?s a way round the repeal? You always have to be optimistic. I fear that it will be a negative decision this time. We have to hope that it will be repealed if that happens, and the people doing that will pay for taking the side of big corporations against the public. I think it?s good that people are speaking out, we?ve been doing that ourselves as well. In the last round, we did that forcefully. In a way, it?s something that we all need to speak out about then follow up closely with the companies. Potentially switching companies if we are ?nding our companies are doing bad things. This is the second very important change that is happening. The ?rst one was the decision to let telcos collect information to the same degree as Facebook and Google, providing, in some ways, more parity between them. That?s the argument, which obviously we don?t buy. I think it should be the other way around: we should restrict Facebook and Google in their collection of information, not giving more people and companies the chance to spy on us. What?s your community approach? It?s the reason why we?re doing Vivaldi in the ?rst place. Before we were doing Opera, we had a very close relationship with our community, and when Opera strayed from the straight and narrow and started doing something totally different, a lot of people were unhappy about that, and we felt we owed them not to leave them stranded. So we started to build a browser, and we?re doing that in very close relationship with our community. We have very different levels; we have volunteers helping us test and give us feedback. We have volunteers helping us to translate. We then have a very active community on https://vivaldi.net, and users are very vocal about what they think we should be doing with the software, what we should be doing better and what we should improve. We take their feedback really seriously. We have this motto, ?We are building this browser for our friends?. It?s not just a saying; we look at our users as our friends, and we want to build a great browser for them. When you think about it, you don?t spy on your friends; you try to keep your friends safe, and you want to listen to your friends? advice?for us it?s real. We want to do the best for our users, it?s also one of the reasons why we?re speaking out about the privacy issues, because we think it?s important, and if you can?t ?x everything on a technical level we can engage in other ways as well. I noticed you had a blogging platform. We?re providing a blogging platform and a webmail service which has no ads, and we?re not going through it, except for viruses and things like that, but it?s part of us providing a platform for our users to hang out and communicate with each other, and tells us what they want and don?t want. Is the webmail separate to the mail client to what you?re working on? Yes, it?s separate. We?re working on the mail client as well. The mail client works with any IMAP service, so it will work with our webmail service, it will work with Gmail even though they do things a bit differently, but any of these other services [...]. We know it?s an important feature for our users, and it?s getting there. I mean, I?ve been personally using it for a long time, but it needs to get to a shape where we feel it?s right to send it out to the users. There are few things we need to ?x there ?rst, but it?s shaping up nicely. You?re also working on a mobile version, is that right? Above Version 1.12 added an Image Properties feature for viewing meta data Top Vivaldi has bookmarks or ?Speed Dials? on the Start Page for accessing your favourite sites When we started, we had mobile as well, and we ran into some technical issues, but we?re working on it. It?s making progress, but it?s still not ready yet. I?m using a very early version myself, but it?s not ready for prime time. What is the most requested feature so far from the community? Mail is very high, but I would say that sync is the highest. We haven?t had synchronisation in yet, and that?s coming very soon. [After the interview, we were advised by Vivaldi that sync would be coming in the next snapshot on 29 November, so it will be available by the time you read this.] That?s also something that we?ve been using internally for quite some time. Part of this is that synchronisation is not only a client side thing, it?s a server side thing as well, so building server technology for that is something that we?ve been doing. It should be ready soon, but we always say ?when it?s ready?. At the time of going to press, Vivaldi was planning to release an experimental build for ARM devices ? including the Raspberry Pi, CubieBoard and ASUS Tinker Board ? on the 5 December. We want the best for our users, it?s also one of the reasons why we?re speaking out about the privacy issues www.linuxuser.co.uk 15 OpenSource Your source of Linux news & views OPINION The kernel column JonMastersreportsonthelatesthappeningsintheLinuxkernelcommunity, asdevelopmentonnewfeaturesfor4.15closeswiththereleaseof4.15-rc1 inus Torvalds announced the release of Linux kernel 4.15-rc1, following the ?usual two weeks of merge window? (the period of time during which disruptive changes are allowed into the kernel tree). In his mail, Linus notes that this is ?about the only thing usual about this merge window?. Due to the US Thanksgiving holiday, many developers (including Linus) were on vacation at the back end, and while Linus had warned everyone to front-load their patch pull requests, he did have to get much more strict about what he was willing to take at the last minute. Overall, he was fairly happy about the process, and in particular ?really liked? enforcing that patches had to have ?owed through Stephen Rothwell?s linux-next tree before going to Linus. This development cycle is expected to be a little different from those in the recent past since it lines up so well with the Christmas holidays. If things go according to plan, 4.15 ?nal will be out in the ?rst week or two of the new year, but we are anticipating a few bumps along the way. We?ll come back to those in a future issue, meanwhile let?s look at some of the new features that are coming to a Linux 4.15 kernel via your favourite Linux distribution. L Jon Masters is a Linux-kernel hacker who has been working on Linux for more than 22 years, since he first attended university at the age of 13. Jon lives in Cambridge, Massachusetts, and works for a large enterprise Linux vendor, where he is driving the creation of standards for energyefficient ARM-powered servers. RISC-V update Linus pulled version 9 of the RISC-V support patches into 4.15. RISC-V is a fully open source computer architecture. An architecture describes the fundamental instructions (machine code) and assembly language that any machine compatible with that architecture is capable of executing. It also speci?es certain behaviours, such as the ?ordering? of memory operations and how interactions take place between multiple processors on a chip. With the combination of an architecture speci?cation, reference hardware, and some skilled experience in porting Linux, it is possible to do what Palmer Dabbelt did taking the lead role in upstreaming support for RISC-V. RISC-V is particularly interesting because it is open source. This means that not only is the architecture speci?cation publicly available (which is common ? multi-thousand page manuals are 16 available online for Intel x86 and ARM, among many other possible architecture choices) but that it is developed in the open using mailing lists (just like any other open source project), and that the reference hardware designs are fully open as well. What does it mean to have an ?open source? chip design? Well, this means that you can download the RTL (hardware description) written in Verilog source code and actually see how every part of the implementation is realised. Using special devices known as FPGAs (?eldprogrammable gate arrays), you can ?synthesize? the RISC-V designs onto a reprogrammable FPGA capable of booting Linux. Then, it is possible to make changes to the design and run those also. For those who aren?t quite aspiring hardware hackers, there are now (relatively inexpensive) RISC-V development boards that provide all of the pieces in a similar fashion to a Raspberry Pi or other single-board computers. Many more such boards will come to market over the next few years, both at the big end (capable of running Linux) and at the much smaller IoT widget end of the spectrum (capable of running an open source IoT OS like the Linux Foundation Zypher project). The RISC-V community is just hitting its stride. They have regular gatherings where new architecture and platform features are debated and worked on, and it certainly feels novel to see a group openly developing what has been traditionally a very secretive process. It?s currently a little more involved to actually run Linux on RISC-V hardware because the initial upstream port doesn?t support much in the way of devices beyond the CPU. As Palmer says, ?While what?s there builds and boots [?] it?s a bit hard to actually see anything happen because there are no device drivers yet. I maintain a staging branch that contains all the device drivers and cleanup that actually works, but those patches won?t be ready for a while?. A question of trust Linus pulled support for AMD?s ?Secure Encrypted Virtualization? into 4.15. This novel feature of recent AMD CPUs allows administrators to create virtual machines that are fully encrypted from the point that data leaves the package of the CPU. Therefore, all memory reads and writes to DRAM chips are performed using encrypted data that cannot easily be decrypted even by very sophisticated and wellfunded actors with many available tools. Since the encryption happens on-chip, an attacker would need to have combinations of tunnelling electron microscopes and the expertise to work around carefully placed physical security precautions on the chip. Thus, this renders moot almost all practical attacks against the new feature. At the same time, SEV opens up many new opportunities for virtual machine hosting companies in far-?ung parts of the globe. Rather than trust that a company hosting a VM is not siphoning off your data, it is possible to have that trust lie between you and AMD as the chip vendor: the hosting company is simply operating the hardware and paying the bills, but has no access to data. RCU work and Linus on time Paul McKenney is a (very) well-known Linux kernel developer and original author of the Linux RCU (read-copy-update) mechanism. RCU allows for socalled ?lockless? updates of certain data structures at runtime by taking clever advantage of the ability for certain algorithms to work with (slightly) out-ofdate values, so long as a careful means is provided to handle when updates become visible. RCU is heavily used in the Linux networking subsystem and in other performance critical code paths. In ?tasks? within the kernel). On a multicore and/or multithreaded system with multiple ?sibling? threads sharing the same LLC (last level cache), it is possible that a call to select_idle_sibling will attempt to wake up more than one task at a time. Those tasks will then race one another, with one of them ultimately winning and the other possibly being migrated (at some cost) onto another CPU. The patch aims to reduce the race window to a few instructions. Benchmarks were inconclusive but showed a modest performance improvement. Marc Gonzalez started a lengthy back and forth between Linus Torvalds and Russell King (the 32-bit ARM subsystem maintainer) about delay handling in the Linux kernel. Marc writes device drivers for embedded systems that are designed within his company (apparently by a guy who sits diagonally across). Like all device drivers, there are periodic requirements for small delays between writing values into hardware ?registers? while the devices process the data and take some action. Linux has a rich framework for handling time, but Marc?s annoyance was that some of these primitives were capable of delaying for less than the time requested, rather than possibly more. This has been a long-standing feature of Linux, and most device driver developers deal with it by adding longer delays. Marc?s point was that this can add up, especially in a ?ash driver of the kind he was developing. There was no resolution, but the discussion resulted in Linus documenting some of his expectations and assumptions around how Linux should be implementing time. RCU allows for so-called ?lockless? updates of certain data structures at runtime addition to authoring RCU, Paul has been working on fully documenting the ?memory model? used by the Linux kernel. Memory models are complex and pretty scary things. They describe how multiple readers and writers within a shared multiprocessing environment see and operate upon shared memory. They?re important because they help (on some level) to ensure that many of the fundamental assumptions programmers have about how programs should operate actually remain true. An RFC (request for comments) patch entitled ?sched: Minimize the idle cpu selection race window? aims to close a long-standing ?race condition? with CPU switching between programs (known as www.linuxuser.co.uk 17 Feature Ultimate Rescue & Repair Kit ULTIMATE RESCUE & REPAIR KIT We can repair it, we have the technology. Neil Bothwick reveals the best tools for diagnosing, ?xing, cleaning and maintaining systems 18 AT A GLANCE �Analyse problems, p20 Look at the evidence to be sure you know what the problem is before trying various ?xes. � Clone & back up, p22 Make a copy of your data, recover it while you can and make sure any attempts at repair do not cause further loss. � Recover your system, p24 What you can do to recover from the problem, with the two main objectives being here can be a certain smugness around some Linux users regarding how their chosen OS is stable, secure and reliable; but stuff happens and it can happen to you. Power failures, disk drive errors, failing memory (the computer, not you) and even ageing PSUs can all cause failures, not to mention faulty or malicious software or plain old ID-TEN-T user errors. There are things you can do to reduce the chances of failure, and other steps to mitigate the damage done when failure occurs, and we will look at some of them, but in the main we?ll be exploring what to do when disaster strikes and how to recover from it as best you can. Ask yourself, what is the most valuable part of your computer system? If your T keeping your data safe and getting your system running again. �Hardware & cleaning, p26 Diagnose faulty or failing hardware and clean up after yourself by getting rid of any packages and ?les that are no longer needed. �Maintenance tips, p28 Keeping your system safe while also maximising your chances of recovery should your computer fail. answer refers to any item of hardware, you are almost certainly wrong. Hardware can be replaced; it may cost a few quid but it can be done and usually easily. That doesn?t always apply to your data: purchased media can be downloaded or bought again, but personal photos, documents and emails can be lost forever. Over the next few pages we will look at the steps involved in diagnosing and ?xing problems, as well as ways of keeping your valuable data safe, both before and after the fact. While some issues can be ?xed from a running system, even if it is running in a crippled way, others require the use of a live CD distro and we?ll be using a couple of these along the way. The live distros are also on this month?s cover disc, a very good reason to keep it safe, but you can also boot them from a USB stick or even over the network if you prefer. It is well known that one of the biggest causes of problems is solutions: ask a plumber how much he earns from clearing up after home ?xes. When things go wrong, we need to take a structured and considered approach to the problem, being sure we know what is wrong before trying to apply, sometimes drastic, ?xes. The good news is that there are plenty of Linux tools to help with this; there are even some that can be used to ?x Windows problems, which we will look at brie?y. There is also a lot you can do to avoid problems cropping up in the ?rst place, from making regular backups to keeping an eye on your system?s health. www.linuxuser.co.uk 19 Feature QUICK TIP Read the man page It you use systemd, read the journalctl man page. It may seem heavy-going at ?rst, but it makes searching the system logs so much more effective. Ultimate Rescue & Repair Kit Analyse problems Before you go about trying to ?x a problem, you will need to be sure precisely what that problem is o what can we do to identify the problem without making it worse? The main thing to remember is ?don?t guess?. Look at what happens when you experience the problem without trying to jump to the ?rst possible cause that pops into your head. Does the computer boot? If not, at what stage does it stop and what messages do you see? Do you see a GRUB menu? Does selecting any option boot the computer? If the answer to any of these is ?no?, reach for your copy of Rescatux which has various options to quickly ?x GRUB issues. Does Linux start to boot but never reach the desktop? Do you get as far as a login prompt in that case? These all help to pinpoint at what stage your system is failing. If you get as far as the distro starting to load services, you have a good chance of ?nding something in the logs. The same applies if the desktop loads but certain programs fail to run correctly. S Use the logs Below Extracting speci?c information from the systemd journal is simple and powerful, once you understand it. Here we are showing errors logged since the last boot, and it looks like we have a problem! Linux logs just about everything: kernel messages go to the kernel buffer while most other programs send information to the system log. If your distribution uses systemd, which most do now, these are combined in the journal. The systemd journal is basically a system log that has the added bene?t of an index, which makes ?nding information much easier. With traditional loggers, the log is written to a plain text ?le which you can search easily enough with grep and other useful utilities. You can still use grep on the systemd journal, but there are better options. This depends on the information being written to the log, and there are two situations where it is not. If The systemd journal is basically a system log but with an index, making ?nding info much easier the system halts early in the boot process, before the ?le system containing /var is mounted and the logger started, nothing can be written to the system log. Some initramfs implementations start the systemd journal very early, buffering the information in memory until it can be saved to disk, so if you can get to a login prompt, you can probably read the journal. If you don?t get a prompt, ?rst try entering single-user mode. At the GRUB boot menu (you may have to hold down a key when booting for it to appear), press E to enter edit mode and add single to the list of options on the kernel line. If you are using systemd, the option is ?systemd.unit=basic.target?; if basic. target fails, try rescue.target. Also delete any options for splash or quiet ? we want to see what the system has to say for itself ? then press Ctrl+X to boot with those options. This should get you to a login prompt in singleuser mode, the most basic of operating modes. Now you can check the system and kernel logs. The system log ?le is usually at /var/log/messages, so read it with: less /var/log/messages Some loggers use a different name, such as /var/log/ current, but it should be one of the most recent ?les in /var/log. Read the kernel log with: dmesg | less Look though these for error messages. Don?t worry too much about warnings: those can be generated by various conditions, like the system looking for something that isn?t there, but warnings are not critical and shouldn?t break things. With systemd, you use: sudo journalctl -b The sudo is necessary if you are not logged in as root as 20 otherwise you may only see the log entries for your user. The -b option tells it to only show entries since the last reboot. You can also ?ne-tune it further by telling it to only show entries marked as error, or more serious, with: QUICK TIP See what went before When you ?nd an error in a log, check the entries immediately before the error: they may give a clue as to what led to it. sudo journalctl -b -p err If you are more comfortable using a pager or grep to work with log ?les (journalctl is much more ?exible but a broken system is not the best time to be learning about something you've not used before), you can send its output to a pipe and it will send plain text, like this: journalctl -b | grep broken Once you see an error, you should look for other messages relating to the same component. With the systemd journal, for example, if you see an error relating to /dev/sda, you can see all messages (system log and kernel buffer) about that drive with: journalctl /dev/sda QUICK GUIDE GRUB options There are a number of options you can add to the kernel line in GRUB to help when diagnosing a problem. First, remove the splash and quiet references so you can see what is going on. If you have already booted with a splash screen and the system just stops booting, press Esc to get rid of the splash screen and, hopefully, reveal something useful. Depending on the initramfs your distro uses, adding rd.shell and rd.debug may help. The former allows the system to drop to a shell if the initramfs fails to complete its handover, while rd.debug logs everything it does. This will be written to a ?le in /run or to the systemd journal if it is running, allowing you to inspect the log or save it to a USB stick for later perusal. Sometimes the info you want to see ?ashes past too quickly for you to read it. You may be lucky and be able to get it back with Shift+PgUp, provided it hasn?t gone too far. Normally, the kernel scrollback buffer holds only a few pages. Sometimes that won?t work: the message you want has gone too far, the scroll isn?t working or you simply haven?t got as far as loading the kernel ? as would happen with a GRUB error. One option is to record the screen with your phone?s camera and play it back in slow motion until you see the information you need. It?s a crude solution but it works. When GRUB is working, it is unusual for anything to go wrong with it. Those errors that do occur are either missing ?le errors after an incomplete update or damage to the ?le system containing /boot. However you get the log information, once you have it you can usually see what is going wrong, and once you know the cause of the problem you are on your way to the solution. The other situation where information is not written to the log is when a program causes an instant crash, meaning nothing can be written to disk before the system goes down. Fortunately, such occurrences are rare and because the crash is instant, you usually know what you did to trigger it. The system log or journal is not the only source of information. For example, if your system boots to Left Turn off the pretty splash screen in GRUB and you can see exactly what is happening on your computer when it boots. All looks well here, but if there is a problem it should be easy to spot When a program causes an instant crash, nothing can be written to disk before the system goes down a console login prompt when you expected to see a desktop, X is probably failing to load. In that case, look in /var/log/Xorg.0.log, in particular at any lines starting with ?(EE)? ? hint: grep EE /var/log/Xorg.0.log There are other logs ?les in /var/log ? not everything uses the system log. Listing the directory in date order immediately after a problem occurs can reveal the most recently written log ?le, which may well hold what you need. ls -lrt /var/log Systemd users can use journalctl -xe to show more detailed information on the most recently logged events. Applying these various techniques, you will either see an error message that you understand or one that baf?es you. In the latter case, try pasting into your favourite search engine, but don?t just jump on the ?rst answer you see ? make sure you read the results carefully to ensure you are trying the right ?x for your situation. www.linuxuser.co.uk 21 Feature QUICK TIP Store them remotely If possible, do not keep backups on the same computer, even if on a different drive. Back up to another computer on your network, or use a cloud storage service. Ultimate Rescue & Repair Kit Clone & back up Make a copy of your data, recover it while you can and make sure any attempts at repair don?t cause further loss f anything is wrong with a disk drive or ?le system, the easiest way to make it worse is to write to it. So the ?rst step is to boot from a rescue CD so you can mount the drive read-only. Before mounting it, you should make a backup image of the drive. The usual tool for this is dd, but that program will exit with an error as soon as it encounters any issue reading from the drive, so we use ddrescue instead. This does the same job as dd but will try again if it hits an error and if it cannot read part of the drive, even after multiple tries, it skips on to the next part. Some of the tools we discuss can be used on an image ?le; in that case, make a copy of the image to work on ? don?t risk corrupting the ?rst image in case you cannot make another one. You invoke ddrescue with three arguments: I ddrescue /dev/sda1 sda.img sda.map Below For extra security, send your backups to another location like an open source cloud server such as Nextcloud The ?rst is the device to read, the second is the image ?le and the third is a map ?le that ddrescue uses to keep track of any blocks it couldn?t read. You can sometimes recover more data by running the command more than once: ddrescue uses the map ?le to know which blocks to try again, although this is more likely to be effective with scratched optical discs than hard drives. Naturally, you can?t store either the image or the map ?le on the drive you are trying to image, so you need to use either Above There are many backup programs available; this one is Dιjΰ Dup. It is less important which one you choose, but essential that you actually use it, preferably on an automated schedule a separate drive, maybe an external one, or a network mount. If your copy stops because it can?t read past a damaged sector of the source drive, run it again with the -R option, which scans the disk in reverse. If there is only one sector that is damaged, this should reconstruct all of the disk apart from this part. If there are multiple occurrences, investigate the -i option that allows you to scan from a speci?c position. The ddrescue manual at http://bit.ly/ddrescuemanual contains a number of useful examples, including one with details on handling this situation. Backups Much grief can be avoided by making regular backups. As well as a means to recover data in the event of a loss, backups are important in the recovery process. There is always the risk of any attempt to repair ?le system damage actually making things worse Whenever there is ?le system damage, there is always the risk of any attempt to repair it actually making things worse. We've already covered ddrescue, but there is also Clonezilla. This can make a complete image backup of a hard drive, including boot sector, partition table, Linux and Windows ?le systems. Clonezilla is a live CD, available from http://clonezilla.org, that can clone a drive to an image ?le, but it can also clone it to another drive. This is a good option if you have a drive 22 that?s failing. Clone the old drive to the new one, then you can check the integrity of the data on the new one without disturbing or writing to the suspect drive. When copying to an image ?le, Clonezilla can store it on another computer using a variety of network sharing and cloud protocols, but it can also use plain old SSH. Backups are also important from a prevention point of view. There are two important points to remember about backups. The ?rst is that humans should never be trusted with doing them. Most backup programs have an option to schedule regular backups; this is how to do it. There are many to choose from. Your author uses Duplicity, a command-line program that can be from cron, but there?s also a GUI front end called Dιjΰ Dup which makes operation much simpler on desktop systems. These can back up to an external hard drive, another computer via SSH or to many cloud services. The other important point about backups is that you must test them. Most backup programs will let you do a test restore to verify the data, but the ?rst time you Most backup programs have an option to schedule regular backups; this is how to do it use it, do a real restore to a different location, just to be sure that the ?les you thought you backed up are actually there and able to be restored. When you have just lost the originals is not the time to ?nd this out. If your data is really important, you should back it up to a different location: you don't want a power surge destroying both the original and backups. With cheap cloud storage and fast internet connection, this is more practical than ever. Which data is important? If you ever have a drive failure without backups, you will realise it all is. QUICK TIP Keep them separate Keeping your OS and data on separate partitions can limit the damage cause by a failure. Losing either your OS or home is bad, but not as bad as losing both. HOW TO Clone a disk using Clonezilla 1 Choose a method Boot from the Clonezilla live CD and choose a cloning method. Here we are sending to an image ?le, but you can also clone to another disk, as long as it is at least as large as the source drive. Other options allow for a network backup to or from a Clonezilla server running on another computer. 3 Beginner or Expert? Choose your level of expertise. Beginner mode makes the dif?cult choices for you and is the right option most of the time. Expert mode enables you to tweak several of the backup settings, which may be good for Clonezilla gurus but just gives everyone else more chances to get it wrong. 2 Storage locations 4 Whole or partition? When backing up to an image ?le, it does not have to be on the same computer. Backing up to a second local drive is fastest, but you can also use a number of network protocols. SSH is the simplest when it can be used, but you can even backup to a web server using WebDAV. Clonezilla can save a whole disk or just a single partition. When saving a whole disk you get the partition tables, bootloader and an image of each partition, plus the info Clonezilla needs to put it all back together. A partition backup is just a compressed image of that partition. www.linuxuser.co.uk 23 Feature QUICK TIP Don?t get frazzled If you are going to mess with cables inside the computer, unplug it ?rst, don't just turn it off. Make sure you ground yourself before touching anything, you don't want to make things worse with a zap of static. Ultimate Rescue & Repair Kit Recover your system What you can do to recover from the problem, keep your data safe and get your system running again isk drive failures are a common cause of grief and can fall into a number of categories. Terminal drive failure is the worst and you can usually say goodbye to your data, unless it is valuable enough to justify the services of a specialist recovery company, but anything that valuable would be backed up, right? Next is a corrupted partition table, where the drive is visible but none of its partitions can be seen. Before you start trying to ?x the drive on these cases, may sure it is at fault. Try a different cable, plug it into a different port, or even computer, to be sure that it is not something as simple as a loose or damaged cable, or a problem with your computer?s disk controller hardware. Naturally, this is more dif?cult with a laptop system but still possible in most cases. Another category is ?le system errors: the partitions are visible in /dev/ but cannot be mounted, or will mount but the data appears garbled. Finally, we have missing ?les when the ?le system is intact but important ?les have been deleted (by someone else of course). Where we go next depends on the type of fault with the drive. If the drive shows up but the partitions don?t, or the partition table is otherwise corrupted, we can rebuild the table. See the instructions at the bottom of the page. D Corrupted file systems If the ?le system mounts but the data or metadata looks wrong, you probably have ?le system corruption, possibly caused by an unclean shutdown. The standard tool for ?xing this is fsck. At its most basic, you invoke: fsck /dev/sda1 Note that you run fsck on a partition, not the whole disk. The fsck program is basically a wrapper that identi?es the ?le system type and then calls the fsck program for that type, for example fsck.ext4. For some ?le systems, the fsck program actually does nothing but returns success. For those, such as XFS, you will need to run the appropriate repair tool. For standard ?le systems like etc3/4, fsck should do the job. It is an interactive program and will prompt for each error or inconsistency it ?nds. These could run initially thousands, so you may want to run fsck with the -y option, which assumes a yes response to most questions, but not the most critical ones. Alternatively, use -p, which ?xes any problems that can be safely repaired without intervention. If your system won?t boot, or you have some sort of hard drive error, you should boot from a live CD to carry out any further work. There are many live Linux CD/DVDs, most of which are generally useful. If you have something like the latest Ubuntu image hanging around, that may well have all the tools you need. However, there are live distros speci?cally designed for use when problems arise, such as SystemRescueCd, HOW TO Recover lost partitions with Rescatux 1 Rebuild GPT partition table The GPT standard stores backup copies of the partition table on the disk. Using gdisk, press R to enter the recovery menu and C to load the backup table. To rebuild the main partition table from the backup, press B, then W to save. 24 2 DOS partition table The old-style DOS partition table is less friendly, with only one copy. So before things go wrong, use fdisk to dump the partition data to a ?le, just in case things go wrong. If you don?t have a backup, you?ll need to try testdisk. 3 Scan for partitions You can use testdisk to scan the entire disk for the tell-tale signs of partitions. Run testdisk from a root terminal to see a list of available drives. You can also pass it the name of an image ?le you created with dd or ddrescue. which boots to a command prompt by default but can also be booted to a rather spartan-looking desktop if you?re more comfortable with that. As is implied by the name, this distro contains many of the tools needed to repair Linux (and Windows) systems. It?s on this month?s cover disc and it would be a good idea to keep this in a safe place. The distro also includes a script to install itself on a USB drive for a more portable solution. Another popular rescue disk is Rescatux. While the range of problems that can be solved with this one is more limited, the one it is designed to ?x can usually be resolved by the click of a button. Its RescApp program contains a number of button to ?x common problems, making it good for a quick ?x. Not all faults are caused by hardware or software; quite a few are caused by users. Hands up anyone who?s never forgotten a password ? they?re the ones with weak passwords. Forgetting a user password is not usually an issue if you can log in as root to change it, but what if you forget the root password, or need to use sudo as on Ubuntu systems? It?s actually remarkably easy: after booting into a rescue CD, mount your root partition at, say, /mnt/root, then change the password with passwd: passwd --root /mnt/root username Rescue CDs log you in as root when they boot, so you can change the password of any user, including root, using this method. Linux to the rescue While using a Linux live CD to ?x a broken Linux system seems entirely logical, using one to ?x Windows problems is not so immediately apparent, but it is possible. The obvious application of this is mounting an unbootable Windows partition in order to recover your documents and other ?les. NTFS support in the Linux has always 4 Choose format You need to know the type of partition table to use. Most distro (and Windows) installers still use the Intel/ DOS format, so select that unless you know otherwise. This is the format testdisk will use to save back any partitions it ?nds. 5 Left It may not be pretty, but Rescatux does make ?xing certain problems, for Linux and Windows, very easy. Click the button, read the help, click Run and all is well! been rather limited, but it does allow read-only mounting of Windows partitions with no issues. Alternatively, you can use ntfs-3g, a FUSE ?le system that allows full read-write access. Given the previous comments about not trying to write to a broken system, this may not be such an advantage in this case, but most distros and live CDs include ntfs-3g and you can mount a Windows partition with: ntfs-3g -o ro /dev/sda1 /mnt/windows This uses the option to mount read-only, which is recommended if Windows did not shut down cleanly. The mount point, /mnt/windows in this example, must exist. Also included with the ntfs-3g package is ntfs?x, which is run with the Windows device as its only argument. It ?xes some NTFS problems, resets the journal and forces a check of the ?le system next time you boot into Windows, but it is not a Linux version of chkdsk. You can also ?x lost Windows passwords. Both SystemRescueCd and Rescatux can do this for you. With SystemRescueCd, go to the system tools submenu at the boot screen. Rescatux makes it easier: there is a set of buttons for Windows operation in RescApp, including password resets and promoting a user to administrator ? both of which can help when you have locked yourself out of Windows. Look for boundaries Analyse is where the work really starts. Testdisk will scan your entire disk looking for signs of partition boundaries. Depending on the partition table corruption, whether there?s a backup and the disk size, this may take a while. 6 Back up info Once partitions are detected, testdisk will offer you the option to back up the current information before going ahead and attempting further recovery attempts. We strongly recommend that you do this! www.linuxuser.co.uk 25 Feature QUICK TIP Get your duster out Keep your computer clean. If it gets dusty on the outside, think what the inside may be like. A couple of minutes with an air duster every so often is better than waiting for problems to strike. Below While smartd monitors your drive?s health in the background, you can use GSmartcontrol to view its state graphically Ultimate Rescue & Repair Kit Hardware & cleaning Diagnose failing hardware and clean up after yourself by eliminating packages and ?les that are no longer needed hile we are looking at the use of software to ?x things, sometimes the problem exists in hardware, so we need to be able to test it. Random crashes or failures are particularly dif?cult to diagnose, but there are a number of things you can look for. Was the computer heavily loaded at the time? Were the fans making more noise than usual? Was there a lot of disk access? Most hardware is really reliable these days, but there are a few weak links. The ?rst is the power supply. Cheap, unbranded power supplies can age badly and cause random failures as voltages ?uctuate. They are also dif?cult to test thoroughly without proper test equipment, but there are a couple of simple tests you can try. If you have a desktop computer with a standard ATX power supply, and a spare available, just swap it out. That is the easiest way to test. Another option is to unplug non-essential hardware, to reduce the load on the PSU. If it has just reached the point where it is causing failures, a slight reduction in load can bring it back into its comfort zone. The next cause of instability is heat. Modern hardware contains multiple temperature sensors, so install a system monitor that lets you keep an eye on them. Some distros use Conky, which can be very attractive. Or you can use GKrellM, which is less attractive but much easier to set up to tell you what you want to know. Computers rely on a good ?ow of air to keep them cool; in the case of laptops, often through small spaces. It is easy for this path to get partially blocked, reducing cooling and increasing temperature. A can of compressed air (often called an air duster) is a good way of clearing things out ? whatever you do, don?t be tempted to use a vacuum cleaner as they can generate enough static electricity to damage your electronics. W QUICK GUIDE Power protection One of the easiest ways to shorten the life of hardware is with dirty power. A decent UPS (uninterruptible power supply) not only allows your computer to shut down cleanly in the event of a power failure, it protects against voltage ?uctuations (brownouts) that can seriously weaken hardware, especially disk drives, and voltage surges that can harm motherboard components and more. One UPS can protect more than one computer. Power supply and cooling problems are relatively easy to ?x, but if left unattended can cause expensive damage. Fluctuating voltages and high temperatures can damage expensive components. A new power supply can even save you money as modern ones are more ef?cient, saving you on your power bills and reducing the overall temperature load. A failing power supply can make the computer more likely to fail when under heavy load, and so can faulty memory. The good news is that this one is easy to test. Many boot discs, including our DVDs, have an option to run memtest86 (or its fork, memtest86+). Memtest86 runs a battery of tests on your memory and reports any errors. As memory failures can be intermittent, you should let it run for at least two or three passes, overnight is ideal. If a failure shows up, you will need to remove individual memory sticks and run the test again to see which one is faulty ? unless you have only one to start with. Getting SMART The other weak link in the hardware chain is the hard drive. While some people worry about the lifespan of SSDs, the truth is that they should last for years with no problems. Hard drives, on the other hand, are complex electromechanical beasts with lots of parts to go wrong. Luckily for us, they are also easy to test thanks to a technology called SMART (Self-Monitoring, Analysis and Reporting Technology) built into most drives. You may need to enable SMART in the computer?s BIOS setup menu, then you can use a program called smartctl to check the drive?s health. Unlike testing memory, most SMART tests run in the background while the drive keeps working as normal. To get a health report on a drive, run: sudo smartctl -H /dev/sda 26 We use sudo because smartctl needs root permissions for low level access to the drive. To run a thorough test on a drive, you would use: sudo smartctl -t long /dev/sda This test can take a while and smartctl should give you an estimate of the time needed. When it is complete, you can see the results with: sudo smartctl --log=selftest /dev/sda And see details of any errors found with: sudo smartctl --log=error /dev/sda This does rely on you running the program regularly and looking at the log output, something we humans are really bad at. Fortunately for us, smartctl has a sister program, smartd, that runs in the background. Just edit /etc/smartd.conf, ?nd the line that contains only DEVICESCAN and change it to: DEVICESCAN -m me@my.email.com Clean A clean, tidy system is less likely to experience problems. We are using two meanings of ?clean? here: free of infection and free of clutter. Computers are good at many things; one thing they do particularly well is accumulate unwanted ?les all over your hard drive. Apart from wasting space, causing fragmentation and increasing backup times, this can have an adverse effect on your system. You can clean out your home directory from time to time, getting rid of those ?les you downloaded but no longer need. There are also temporary ?les that various programs save to your disk: things like browser cookies, cached ?les and various other cruft spread around. Help is at hand in the form of BleachBit (www.bleachbit.org). This program scans your home directory and identi?es ?les that you should no longer need. It can take a while to complete its magic, then it shows you a list of what it has found and you can choose which ?les it should delete. And cleaner Having unnecessary programs installed is also a potential risk to your system. Those programs you installed to try out then never used again are still there, and some can present a security risk. Uninstalling unwanted programs is a start, but often when you install a program, it pulls in some dependencies. With Debian/Ubuntu-based systems, you can get rid of dependencies that are no longer needed with: sudo apt-get autoremove Above BleachBit can ?nd an amazing amount of cruft, but make sure you really don?t need any of these ?les before letting it delete them Keeping clean from infection is even more important. While Linux is considered by many to be immune to virus contamination, that is not true. While there has been little Linux malware, there has been some. If you share ?les with Windows systems, you also need to be aware of the possibility of transferring Windows viruses, which are a far more common beast. While Linux has very little in the way of viruses, it does have a good virus checker, ClamAV. ClamAV should be in your distro?s repositories and consists of a number of programs. The While Linux has very little in the way of viruses, it does have a good virus checker, ClamAV ?rst you really need is freshclam, which updates the virus databases ? run this before scanning, or from a cron job. You should then run clamscan to check for viruses, giving it the directory to search, for example: clamscan --recursive --cross-fs=no / The recursive option descends into subdirectories, so this would scan the whole ?le system. Disabling cross-fs prevents it searching other ?le systems, like /proc or any network ?le systems. It also means that if, for example, /home is on a separate partition, it will another invocation of clamscan. If your computer is always on, it is easier to drop a two-line script into /etc/cron.daily that calls freshclam and clamscan to check everything every night. Left Testing memory with memtest86+. With a name like that, what else would you use it for? It is important to let it run for several passes ? overnight is usually best www.linuxuser.co.uk 27 Feature QUICK TIP Don?t get a complex Keep your computer safe, but don't get hung up on it. Remember to spend most of your time using it for what you bought it for. Ultimate Rescue & Repair Kit Maintenance tips Keep your system safe and prepared for problems to maximimise your chances of recovery should it fail nce everything is working well after you have repaired it, or preferably before you have needed to, there are things you can do to keep your system in good health, or at least be better prepared for problems. The ?rst, and most important, is to set up a backup system, and make it an automated one. Don?t rely on your remembering to do it, but do test the backups from time to time. While you are automating the backups, set up smartd to run in the background and have cron run Rootkit Hunter. Once those are set up, you can forget about them: they will let you know if they want your attention. O Right Conky is a highly con?gurable system monitor that enables you to keep an eye on many aspects of your system?s health and status To keep an eye on your temperatures: a desktop system monitor like Conky or GKrellM can help here, showing you immediately if things start to get warm. You will naturally see temperature increases when your computer is working hard, transcoding a video or compiling software, but if it starts to rise in normal use you should investigate immediately. High temperatures can kill hardware, but don?t wait for them to get too high: more modest rises can shorten the life of components. Laptops in particular can suffer from borderline cooling ? are the fans louder than before? In most cases it is simply a matter of cleaning dust from around cooling vents. Tidy up after yourself If you often try different software packages, remove the ones you opt not to use. The more software installed, the greater the chance of exposing your system to a security vulnerability. Similarly, keep your system up to date: most updates to software are to ?x bugs or security. Try to keep to packages from your distro?s main repositories as much as possible, as they are tested for compatibility. Adding loads of PPAs may seem a good idea, but you may (not will, only may) end up with a less stable system. Never use the power switch to shut down your system. 28 QUICK GUIDE The magic of SysReq Linux is usually stable enough to survive misbehaving software, but it is possible for a program to lock up the whole computer. Before you reach for the Power or Reset button and risk corrupting your ?le systems, there is a better way to get out of it. You can send commands directly to the kernel by holding down the Alt and SysReq (aka PrtScr) keys and pressing certain letter keys. As the kernel listens for these directly, they work even if X is completely locked and accepting no input. The keys normally used to get out of a lockup are R, to reset the keyboard ? this occasionally ?xes the problem on its own. Next, press E to send a TERM signal to all processes, asking them to shut down cleanly, writing any data to disk and closing any open ?les. Next in line is I, which sends a KILL signal to all remaining processes, forcing them to shut down. Pressing S tells the kernel to sync, ?ushing all buffers to disk so that remaining open ?les can be closed cleanly. Press U to unmount all ?le systems and remount them read-only, to avoid further data corruption. Finally, B reboots the system. So, that is: hold down Alt and SysReq and press R-E-I-S-U-B in turn, preferably leaving a couple of seconds between each. There are several, mainly silly, mnemonics to help remember that sequence, the most appropriate being Reboot Even If System Utterly Broken, but the easiest way to remember the sequence is that it is BUSIER backwards. This is not something you should need very often, but it is well worth remembering when you do. If your desktop locks up and appears unresponsive, it may still be possible to reboot your computer cleanly ? cutting the power is asking for a corrupted ?le system. If it is only the desktop that is locked up, and you?re running an SSH server, try logging in from another computer and issuing the shutdown command from there (or systemctl reboot on a systemd distro). If all else fails, try the magic SysReq key combinations: these operate at the kernel level and can help you shut down cleanly when nothing else responds (see The Magic of SysReq box for details). Most importantly, though, don?t panic. Most of the time your system will just work. If it should suddenly hit problems, there is almost always something you can do about it, and it should only involve some common sense and detective work, not a credit card. ON SALE NOW! AVAILABLE AT WHSMITH, MYFAVOURITEMAGAZINES.CO.UK OR SIMPLY SEARCH FOR T3 IN YOUR DEVICE?S APP STORE SUBSCRIBETODAYANDSAVE! SEE WWW.MYFAVOURITEMAGAZINES.CO.UK/T3 Subscribe Never miss an issue FREE DVD TRY 2 UBUNTU SPINS www.linuxuser.co.uk THE ESSENTIAL MAGAZINE FOR THE GNUGENERATION ULTIMATE RESCUE & REPAIR KIT � Digital forensics � Data recovery � File system repair � Partitioning & cloning � Security analysis IN-DEPTH GUIDE The future of programming The hot languages to learn £5.19 per issue PAGES OF GUIDES > MQTT: Master the IoT protocol > Security: Intercept HTTPS > Essential Linux: The joy of Sed ALSO INSIDE ISSUE 186 PRINTED IN THE UK £6 49 » iS Storage DiskAshur Pro2 » Ja ava: Spring Framework » Disaster relief Wi-Fi Subscribe an * > MQTT: Maste ter the IoT protocol DES 20% Every issue, delivered straight to your door PAG GES OF to learn The hot languages progra amming T e fu uture of Never miss an issue ue 13 issues a year, and you?ll be sure to get every single one Delivered to y yo our u om alysis Free delivery of direct to your y issue, i step What our readers are ?I?ve only just found out about this magazine today. It?s absolutely brilliant and exactly what I was looking for. I?m amazed!? Donald Sleightholme via Facebook 30 ?@LinuxUserMag just e by pos Wow what a fantastic is ssue! ssue! e! I was just just a about to start playing witth mini-pcs and a soldering iron n. TY? T @businessBoris via a Twit T tter tt UNTU SPINS the biggest savings s Get yourr favourite magazine for f less b byy ordering direct ut us? nks for a great magazine. I?ve been a re egular subscriber now for a number of years.? Matt Caswell via email Pick the subscription that?s right for you MOST FLEXIBLE GREAT VALUE Subscribe and save 20% One year subscription Automatic renewal ? never miss an issue Great offers, available worldwide One payment, by card or cheque Pay by Direct Debit 3FDVSSJOHQBZNFOUPGbFWFSZTJYNPOUIT TBWJOHPOUIFSFUBJMQSJDF Name of bank Instruction to your Bank or Building Society to pay by Direct Debit "TJNQMFPOFPGGQBZNFOUFOTVSFTZPVOFWFS NJTTBOJTTVFGPSPOFGVMMZFBS5IBUTJTTVFT EJSFDUUPZPVSEPPSTUFQ Originator?s reference UK b TBWJOH PO UIF SFUBJM QSJDF 7 6 8 1 9 Europe ?88.54 USA $112.23 Rest of the world $112.23 Pay by card or cheque Address of bank Pay by Credit or Debit card Mastercard Visa Amex Card number Account Name Postcode Expiry date Sort Code Account no Pay by Cheque 1MFBTFQBZ'VUVSF1VCMJTIJOH-UE%JSFDU%FCJUTGSPNUIFBDDPVOUEFUBJMFEJOUIJTJOTUSVDUJPOTVCKFDUUPUIF TBGFHVBSETBTTVSFECZUIF%JSFDU%FCJUHVBSBOUFF*VOEFSTUBOEUIBUUIJTJOTUSVDUJPONBZSFNBJOXJUI 'VUVSF1VCMJTIJOH-UEBOEJGTPEFUBJMTXJMMCFQBTTFEPOFMFDUSPOJDBMMZUPNZ#BOL#VJMEJOH4PDJFUZ #BOLT#VJMEJOH4PDJFUJFTNBZOPUBDDFQU%JSFDU%FCJUJOTUSVDUJPOTGPSTPNFUZQFTPGBDDPVOU Signature I enclose a cheque for Date £ Signature Made payable to Future Publishing Ltd Date Your information Name Address Telephone number Mobile number Email address Postcode Please post this form to Q 1MFBTFUJDLJGψZPVXBOUUPSFDFJWFBOZDPNNVOJDBUJPOTGSPN'VUVSFBOEJUTHSPVQDPNQBOJFTDPOUBJOJOHOFXT TQFDJBMPGGFSTBOEQSPEVDUJOGPSNBUJPO Linux User & Developer Subscriptions, Future Publishing Ltd, 3 Queensbridge, The Lakes, Northampton, NN4 7BF, United Kingdom Order securely online www.myfavouritemagazines.co.uk/sublud Speak to one of our friendly customer service team Call 0344 848 2852 These offers will expire on 31 January 2018 Please quote code LUDPS17 when calling *Prices and savings are compared to buying full priced print issues. You will receive 13 issues in a year. You can write to us or call us to cancel your subscription within 14 days of purchase. Payment is non-refundable after the 14 day cancellation period unless exceptional circumstances apply. Your statutory rights are not affected. Prices correct at point of print and subject to change. Full details of the Direct Debit guarantee are available upon request. UK calls will cost the same as other standard ?xed line numbers (starting 01 or 02) are included as part of any inclusive or free minutes allowances (if offered by your phone tariff). For full terms and conditions please visit: bit.ly/magtandc Offer ends 31 January 2018. Tutorial Essential Linux PART FIVE Master shell scripting: The Joy of Sed John Gowers is a university tutor in Programming and Computer Science. He likes to install Linux on every device he can get his hands on, and uses terminal commands and shell scripts on a daily basis. Resources A terminal running the Bash shell (standard on any Linux distro) A ?dictionary? text ?le, containing a line-separated list of words in the English language (check the /usr/ share/dict and Learn the many uses of the stream editor sed, an extremely versatile command for text replacement In the ?rst part of this series, we learned about the UNIX philosophy that underpins all the commands we have been learning about. Part of that philosophy was the insistence that all programs should communicate with one another through a text stream, since that is a universal interface. This is a powerful idea: as long as two programs function by taking in textual input and printing out textual output, then they can interact together through a pipeline, even if they haven?t been explicitly designed to work together. The main pitfall behind this idea is that the textual output from one program might not be in the format that the second program expects as input. In the last issue, we met one of the tools that can be used to get round this program: the program grep ?lters out lines of output depending on whether they ?t a particular pattern. In this issue, we will meet a more powerful program: the ?stream editor?, or sed, that can act as the glue between different programs. Rather than ?ltering lines based on a pattern, sed can actively change a line The most important sed command, and the one we will use most frequently, is s, which is short for ?substitute?. The basic syntax is very simple: $ <<< "couscous" sed 's/cous/pom/g' pompom In this example, the s at the beginning is the command, substitute, followed by a slash /. This is followed by the string to remove, cous, followed by another slash / and lastly by the string to replace it with, pom, followed by a /\n/ !G olleH\n$ s/\(.\)\(.*\n\)/&\2\1/ olleH\nlleH\no$ // D lleH\no$ #!/bin/sed -f s/\(.\)\(.*\n\)/&\2\1/ lleH\nleH\nlo$ // D leH\nlo$ /\n/!G s/\(.\)\(.*\n\)/&\2\1/ //D s/.// s/\(.\)\(.*\n\)/&\2\1/ leH\neH\nllo$ // D eH\nllo$ s/\(.\)\(.*\n\)/&\2\1/ eH\nH\nello$ // D H\nello$ s/\(.\)\(.*\n\)/&\2\1/ H\n\nHello$ // D \nHello$ s/\(.\)\(.*\n\)/&\2\1/ \nHello$ // D \nHello$ s/.// Hello$ directories or search online) 32 s is for substitute olleH /var/lib/dict The wget program (see your package manager or download from www.gnu.org/ software/wget/) of output through applying substitutions. Sed is the precursor of more powerful stream editors such as AWK, Perl and Python; it is very well suited to simple text-processing tasks. For example, we will later see that we can use sed to translate the output of a package manager program into input accepted by the rm program. third slash /. Last, the letter g is a supplementary option to the s command: it tells s to apply the substitution throughout the line. If we leave off the g, then sed only applies the substitution once per line: $ <<< "couscous" sed 's/cous/pom/' pomcous The power of the s command comes from the fact that the ?rst argument can be an arbitrary regular expression, rather than just a string. For example, we can write a Sed script that will delete all non-alphanumeric characters from input: $ sed 's/[^A-Za-z0-9 ]//g' It is time to eat, grandpa. It is time to eat grandpa If we want to use the Extended Regular Language, then we can use the -E switch, just as with grep: $ sed -E 's/[aeiouwy]+/oodle/g' Do it! Now! Doodle oodlet! Noodle! Here, the extended regular expression [aeiouwy]+ matches any sequence of vowels (or ws or ys). Grouping with sed In the last issue, we learned that we can group parts of a regular expression together using brackets preceded by backslash (\(...\)) or, in the Extended Regular Language, plain brackets ((...)). grep then allowed us to refer back to these groups using numerical codes \1, \2 and so on. For example, the following regular expression matches a word, followed by a second word, followed by the ?rst word: $ grep -E '([^ ]+) [^ ]+ \1' hand in hand hand in hand hand in glove Since we have enclosed the pattern [^ ]+ corresponding to the ?rst word in brackets, we can refer to it later on as \1. We can use exactly the same notion of grouping for our regular expressions with sed. Backreferencing of groups is even more powerful in sed, however, since we can also refer to the group on the right of the s command. This allows us to refer to replace a particular string with a Text-processing programs such as sed are the glue between different command-line programs FLAGS TO THE S COMMAND g Apply the substitution wherever it appears in the line 3 (or another number). Apply the substitution the 3rd time it appears in the line p Print out the substituted string an extra time (often used with the -n ?ag to suppress automatic printing) w ?lename e Print the substituted string to the speci?ed ?le Treat the substituted string as a Linux command and replace the string with the output of that command second string that is derived from the ?rst. For example, we can write a sed script to reverse the ?rst two words in each line: $ sed -E 's/([^ ]+) ([^ ]+)/\2 \1/' is sed worth learning sed is worth learning As with grep, the numbers 1, 2 and so on refer to the order in which the bracketed expressions appear in the matching regular expression. In this case, we want to print the second group ?rst and then print the ?rst group. A related expression in sed is the ampersand &. If we use this on the right-hand side of an s command, then it refers to the expression that was matched on the left: $ sed -E 's/([^aeiou ]?)([^ ]*)/& in pig latin is \2-\1ay./g' hello hello in pig latin is ello-hay. apple apple in pig latin is apple-ay. Chaining sed commands Sometimes, we might want to run more than one sed command over our program in sequence. Sed allows us to chain commands together using the semicolon character ';': $ sed 's/dog/dawg/g;s/cat/katt/g' It's raining cats and dogs. It's raining katts and dawgs. Note that each command is run over the string in turn. So the command s/cat/dog/g;s/dog/cat/g will not swap the words cat and dog: it will ?rst replace all instances of cat with dog and then replace all instances of dog with cat. The result is that the original instances of cat are changed back to cat again. Sed scripts Just as we wrote Bash scripts before, we can write sed scripts, which can be run as standalone programs. Above The s command provides many more ?ags than the ones we have met. To use these, put them at the end of the command. For example, s/dog/cat/3 Escape characters and single quotes As we learned in part two of this series, texts in single quotes are treated exactly as they are written, whereas strings in double quotes are subject to various manipulation rules. It might seem surprising that, although sed supports escape characters such as \n for a new line, sed scripts are almost always written to the command line in single quotes. The reason for this is that it is sed itself that interprets the escape characters: if we put the string in double quotes, they?d be interpreted by Bash and would then not work correctly with sed. www.linuxuser.co.uk 33 Tutorial Further directions The commands we have learned are quite simple, but sed supports further commands that allow us to perform looping, saving of state and operations on multiple lines at a time. If you want to learn about these, a good place to start is by typing man sed or (better!) info sed at the command line. The sed script in the main image is an emulation of the rev command, which reverses its input. If you want to understand some of the more advanced features of Sed, a good place to start is by trying to convince yourself why this script works as it does. Essential Linux To write a sed script, we start off with the following shebang line: #!/bin/sed -f ?which tells the shell that the script should be run by the sed command. If we want to use the Extended Regular Language, we should use #!/bin/sed -Ef instead. We can then list the sed commands one by one. For example: #!/bin/sed -f # Remove spaces s/ //g # Replace 'gobi' with 'tuna' s/gobi/tuna/g Save this into a ?le called luad-replace.sed, and then run chmod +x luad-replace.sed. You can then run the script directly from the command line: $ ./luad-replace.sed forgo bite fortunate Note that we used comments in the script above. In a sed script, any ?le starting with a hash sign # is treated as a comment and ignored by sed. Editing files with sed It is easy to use the normal pipeline to run a sed command over a ?le. For example: $ <email.txt sed 's/damn/darn/' > clean_ email.txt However, this requires us to create a new ?le to hold the result. Sed allows us to modify a ?le in place using the -i switch: $ sed -i.orig email.txt 's/damn/darn/' The .orig after the -i switch is optional, but useful. If we make a mistake with our sed command, then we could end up losing our original ?le. If we add the optional extension after -i, then it will save a copy of the original Right Sed scripts are used in many major pieces of software. This one is part of Bash itself, used for printing quotation marks 34 ?le to email.txt.orig, just in case we need to restore it again. Selective printing: a Case Study The package manager that your author uses to install software on his computer sometimes prints out error messages of the form: ... Installing... ?λ??λ????λ?????λ?????????λ???λ???????? ??????????? ?λ??λ????λ?????λ??????????λ???λ???????? ??????????? ... Installation failed! This happens when a package tries to install itself by overwriting ?les that already exist. In this case, the package manager will not overwrite the ?les, but will pass control back to the user so that they can decide what to do with the ?le con?icts. More often than not, it turns out that the ?les are left over from a broken installation and should be deleted. Sed provides many pieces of functionality besides simple replacement of text In order to delete these packages automatically, it helps to have the names of the offending ?les printed out line by line. Unfortunately, the package manager prints out a lot of other information besides the paths to the ?les. We can use a sed script to print out just the ?lenames: $ sudo pacman -S package-name 2>&1 | sed -E 's/package-name: ([^ ]*) already ?????????????????????' ... Installing... ??λ????????? ??λ?????????? ... Installation failed! This is a good start, but it still prints out input from the package manager that we do not want (for example, the text Installing...). We want to print out only those lines that match the pattern, and then apply the substitution. One approach would be to pipe the output through grep before piping it through sed: $ sudo pacman -S package-name 2>&1 | grep 'package-name: [^ ]* already exists in ?????????' | sed -E 's/package-name: ([^ ]*) already ?????????????????????' But this requires us to type the pattern twice. Luckily, it turns out that we can do the whole thing with one sed command: $ sudo pacman -S package-name 2>&1 | sed -E -n 's/package-name: ([^ ]*) already ??????????????????????' ??λ????????? ??λ?????????? ... The option -n to Sed suppresses automatic printing. If we use -n, then Sed will not print anything out unless we tell it to. The p at the end of the s command is a command that explicitly tells sed to print out the result after the substitution has been applied. Since we do not apply the substitution to lines that do not match the pattern package-name: [^ ] already exists in ?????????, then these lines will not be printed out. We can pipe the output from this pipeline through xargs rm in order to delete all of the ?les. Address ranges Besides s, there are several other commands that are often used. These commands are typically preceded by an address range, which speci?es which lines to apply the command to. An address range is either a single line, a range of lines or a regular expression. To illustrate these, we will use the p command which we have already met and the command seq 10, which prints out the numbers from 1 to 10. $ seq 10 | sed -n '7p' 7 Preceding the command p with a single line number means that the command is applied only to that line number. In this case, only the number 7 gets printed out. $ seq 10 | sed -n '4,6p' 4 5 6 In the second example, the range 4,6 speci?es that we print out lines 4 to 6 of the output. $ seq 10 | sed -n '/../p' 10 This time, we specify that we should only print those lines that match the regular expression ... To distinguish it from other commands, the regular expression should be delimited by slashes /.../. We can also use regular expressions in range addresses: $ seq 10 | sed -n '1,/[3-5]/p' COMMON SED COMMANDS # Treat the command as a comment q Immediately quit Sed, printing out the current substitution d Delete the current line and move to the next line of input p Print out the current line n Skip the current line (print it out without applying any substitutions) {} Sed treats commands in curly brackets as a group. If we precede a group with an address range, it will be applied to all commands in the group 1 2 3 Here, sed prints out lines from line 1 up till the ?rst line that matches the regular expression [3-5]. This time, the third line of output matches that expression, and all subsequent output is suppressed. Last, we can perform negative matches by appending an exclamation mark ! after the range address: Above Sed supports many commands, but the most commonly used (apart from s) and most simple are the ones in this table $ seq 10 | sed -n '2,10!p' 1 Here, the command prints out all lines except those in the range 2 to 9. Note that we can simulate the behaviour of grep by writing sed -n '/regexp/p' instead of grep regexp. Deleting lines Besides s and p, we have access to a number of other useful commands that are available in sed. Like p, these can all be preceded by an address range to specify which lines they should be applied to. We can use the command d, for example, to delete a line and move to the next line of input without printing anything. For instance, we can run a sed command to remove the small print at the start of the Moby Dick ?le that we downloaded in our very ?rst article for this series [Tutorials, p36, LU&D181] : ? ???? ???????????????????????????????????? old/moby10b.txt >/dev/null $ sed -i moby10b.txt '1,/END THE SMALL PRINT/d' This sed command will delete all lines from line 1 up to and including the ?rst line it ?nds containing the text END THE SMALL PRINT. www.linuxuser.co.uk 35 Tutorial Sysadmin Essential admin commands 20 terminal commands that all Linux web server administrators should know blindfolded Adam Oxford runs South African tech news site www.htxt.co.za. He learned many of these lessons the hard way. Resources Terminal & editor All Linux distros have a terminal and most have a number of terminal-based text editors you can access on the CLI Above Can?t remember that really clever thing you did last week? The history command is your friend Are you an ?accidental admin?? Someone who realised, too late, that they were responsible for the workings of a Linux server and ? because something has gone wrong ? ?nds themselves lost in a world of terminals and command lines? What is SSH?, you may be asking yourself. Do those letters after ?tar? actually mean anything real? How do I apply security patches to my server? Don?t worry, you?re not alone. And to help you out, we?ve put together this quick guide with essential Linux commands that every accidental admin should know. Becoming an accidental admin While we?d argue that they should, not everyone who starts using Linux as an operating system does so through choice. We suspect that most people?s ?rst interaction with Linux happens somewhat unwittingly. You click a button on your ISP?s account page to set up a personal or business web server ? for a website, email address or online application ? and suddenly you?re a Linux admin. Even though you don?t know it yet. When starting out with your web server, things are usually straightforward. Nearly all hosting providers give you a web interface such as cPanel or Plesk to manage your server. These powerful pieces of software give quick and easy access to logs, mail services and one-click installations of popular applications such as WordPress. But the ?rst time you have to do something that isn?t straightforward to do through the graphical control panel, you?re suddenly out of the world of icons and explanatory tooltips and into the world of the text-only terminal. To make things worse, for a lot of people the ?rst time they have to deal with the terminal is when something has gone wrong and can?t be ?xed through the control panel. Or perhaps you?ve just read that there?s a major 36 security ?aw sweeping the web and all Linux servers must be updated at once. Suddenly you realise that your nice control panel hasn?t actually been updating your server?s operating system with security patches and your small personal blog may well be part of a massive international botnet used to launch DDOS attacks against others. Not only are you a stranger in a strange land, you?re probably trying to recover or ?x something that was really important to you, but which you never gave much thought to while it was being hosted for a couple of pounds a month and seemed hassle-free. You are an ?accidental admin?. Someone who is responsible for keeping a Linux web server running and secure, but you didn?t even realise it. You thought all that was included in the couple of pounds a month you pay to your ISP ? and only found out it?s not when it was too late. Since most web servers are running Ubuntu, this guide is based on that particular distribution. And all the commands here are just as applicable to a Linux desktop as they are to a web server, of course. 01 sudo The most fundamental thing to know about Linux?s approach to administration is that there are two types of accounts that can be logged in: a regular user or an administrator (aka ?superuser?). Regular users aren?t allowed to make changes to ?les or directories that they don?t own ? and in particular this applies to the core operating system ?les, which are owned by an admin called ?root?. Root or admin privileges can be temporarily granted to a regular user by typing sudo in front of any Linux command. So to edit the con?guration ?le that controls which disks are mounted using the text editor, nano, you might type sudo nano /etc/fstab (we really Above Even if someone copies your key, they?ll still need a password to unlock it Root or admin privileges can be temporarily granted to a regular user by typing sudo in front of any Linux command don?t recommend this unless you know what you?re doing). After entering sudo, you?ll be asked for your user password. On a desktop PC, this is the same one that you use to log in. If you?re logging into your own web server, however, there?s a good chance that you?ll already be the root user and won?t need a password to make important changes. If you can?t execute sudo commands, your web host has restricted your level of access and it probably can?t be changed. User accounts can be part of ?groups? in Linux and only members of the sudoers groups can use the sudo command to temporarily grant themselves admin privileges. 02 su 03 ifcon?g While sudo gives you great power, it still has limitations. Most of all, if you?ve got a whole bunch of commands to enter, you don?t want to have to type it out at the start of every single line [at least the password has a 5 minute timeout ? Ed]. This is where su comes in, which will give you superuser powers until you close the terminal window. Type sudo su followed by your password, and you?ll see the prompt change from yourname@yourserver to root@yourserver. You might think su stands for superuser, but it?s actually a command to change to any user on the system and if it?s used without an account name after it, su assumes you want to be root. However, using su myname will switch you back to your original, non-super, login. Since you?re troubleshooting a web server, it?s probably a good idea to get as many details about its actual connection as possible noted down. The ??????? command can be run without sudo privileges and tells you details about every live network connection, physical or virtual. Often this is just for checking your IP address, which it reports under the name of the adaptor, but it?s also useful to see if you?re connected to a VPN or not. If a connection is described as eth0, for example, it?s an Ethernet cable; meanwhile, tun0 is a VPN tunnel. 04 you?ll probably need to put sudo in front of anything you chown, but the syntax is simple. An example might be ????? ???λ?????????? ????λ??????. 05 service restart 06 ls No, we?re not telling you to ?try turning it off and on again?, but sometimes it?s a good place to start (and sometimes it?s essential to load changes into memory). It?s possible you might be used to start and stop background processes on a Windows desktop through the graphical System Monitor or Task Manager in Windows. However, in the command-line terminal to a server it?s a little more tricky, but not by much. Confusingly, because many Linux distributions have changed the way they manage startup services (by switching to systemd), there are two ways of doing this. The old way, which still works a lot of the time, is to just type service myservice restart, preceded with sudo when it?s necessary. The new, correct, way is a little more verbose: ?????????????λ???????????????????. So if you want to restart Apache, for example, the core software which turns a mere computer into a web server, the command required would be ????????????? ????λ??λ?λ????????????. The key to understanding the console is all in the path, shown to the left of the command prompt, which tells you whereabouts you are in the folder structure at any given time. But how do you know what else is in your current location? Easy: you use ??. The ?? command lists all the ?les within the folder that you?re currently browsing. If there?s a lot of ?les to list, use ??????? to pause at the end of each page of ?lenames. chown There?s tons more you can learn about chmod (see box, p33) and we strongly recommend that you do, but it has a sister command that?s even more powerful. While chmod dictates what users who aren?t the owner of a ?le can do, the chown command changes the ?le owner and group that it belongs to completely. Again, Above Unless you can read 1,000 lines a second, you?ll need to use ??????? to explore folders 07 cat A command you?ll often see if you?re following instructions you?ve found online ? and aren?t always sure what you?re doing ? cat is short for ?concatenate? and is used to combine ?les together. In its simplest form it can be used to take ?le1.txt and ?le2.txt and turn them Recursive If you are changing names, permissions or ownership, most commands have a -R or -r option, which stands for ?recursive?. Essentially, this changes the attributes of all ?les inside a folder, rather than just the folder itself. www.linuxuser.co.uk 37 Tutorial Sysadmin running, and sudo apt-get upgrade will download and install them. For the most part these are safe commands to use and should be run regularly ? but occasionally updating one piece of software can break another, so make a backup ?rst? 11 grep 12 top 13 kill, killall Using top, you can ?gure out which application is 14 w As computer commands go, there are few more fantastically named for the newcomer than grep [it?s a real verb! ? Ed]. How on earth are you ever going to master this Linux stuff if it just makes words up? But grep is a great utility for looking for patterns within ?les. Want to ?nd every line that talks about Cheddar in a book about cheeses? grep "cheddar"???????????????? will do it for you. Even better, you can use it to search within multiple ?les using wildcards. So grep "cheddar"????? will ?nd every text ?le in which Cheddar is referenced. So now you grok grep, right? Above Nano isn?t the only terminal text editor, but it?s the easiest to use into ?le3.txt, but it can also be combined with other commands to create a new ?le based on searching for patterns or words in the original. Quite often you?ll see cat used simply to explore a single ?le ? if you don?t specify an output ?lename, cat just writes what it ?nds to the screen. So online walkthroughs often use cat as a way of searching for text within a ?le and displaying the results in the terminal. This is because cat is non-destructive ? it?s very hard to accidentally use cat to change the original ?le, whereas other commands might do. 08 ?nd A useful and underused command, ??? is pretty self-explanatory. It can be used to ?nd stuff. Typing it by itself is much like ??, except that it lists all the ?les within subdirectories of your current location as well as those in your current directory. You can use it to search for ?lenames using the format ?????λ??"????λ??????". By inserting a path before the -name option, you can point it at speci?c starting folders to speed things up. By changing the -name option you can search by days since last accessed (-atime) or more. 09 df Maybe your server problems are to do with disk space? Type df and you?ll get a full breakdown of the size and usage of every volume currently mounted on your system. By default it?ll give you big numbers in bytes, but if you run df -h (which stands for ?human readable?), the volume sizes will be reported in megabytes, gigabytes or whatever is appropriate. 10 apt-get update && upgrade Probably the single most important command to know and fear. We all know that to keep a computer system secure you need to keep it updated, but if you?ve got control of a Linux box the chances are that it isn?t doing that automatically. A simple sudo apt-get update will order your system to check for the latest versions of any applications it?s 38 When you?re working in a graphical user interface such as a Linux desktop environment or Windows desktop, there?s always an application like System Monitor or Task Manager which will call up a list of running applications and give you details about how many CPU cycles, memory or storage they?re using. It?s a vital troubleshooting tool if you have a program that?s misbehaving and you don?t know what it is. In a similar way, you can bring up a table of running applications in the Linux terminal that does the same thing by typing top. Like a lot of command-line utilities, it?s not immediately obvious how you can close top once you?re ?nished with it without closing the terminal window itself ? the almost universal command to get back to a prompt is Ctrl+C. using all your CPU cycles, but how do you stop it without a right-click > End process menu? You use the command ???? followed by the process name. If you want to be sure and kill every process with a name that contains that application name, you use ????λ??. So ?????????? will close down a web browser on a Linux desktop. From the weirdness of grep to the elegance of the w command, a whole command in a single letter. If you think another user is logged into your system, this is an important command to know. You can use w to list all currently active users, although don?t rely on it too much as it?s not hard for a hacker to stay hidden. To be sure and kill every process with a name that contains that application name, you use killall Above Keep an eye on the directory path in front of the command line to ?gure out where you are 15 used to remove or delete a ?le, while cp will copy ?les and folders. Just as with the cd command, you can either enter a ?lename to operate on a ?le in the directory you?re working in or a full path starting from the root of the drive with ~. For mv, the syntax is ???????λ?????????????? ?????λ?????????λ????. The big thing to remember is that in the terminal there?s no undo or undelete function: if you rm a ?le it?s gone forever (or at least will require very specialist skills to retrieve) and in a similar fashion, if you mv or cp a ?le you?d better make a note of where it went. passwd You must use passwd with extreme care. Ultra extreme care. Because the next word you write after it will become your login password, so if you type it incorrectly or forget it, you?re going to ?nd yourself in serious trouble. You can only change your own user?s password by default, but if you grant yourself sudo powers you can change any user?s credentials by including their username after the password itself. Typing sudo passwd, meanwhile, will change the password for root. 18 nano If you grant yourself sudo powers, you can change any user?s credentials 19 history Check out the manual (man passwd) page for some useful options to expire passwords after a certain period of time and so on. 16 cd 17 mv & rm & cp If you have a graphical interface and ?le browser, it?s pretty easy to move to new locations on your hard drive just by clicking on them. In the terminal, we know where we are because of the path (shown to the left of the command prompt), and we switch location using cd, which stands for ?change directory?. The cd command in mainly used in three ways: � ?????????λ?? ? This will move you to that folder, provided it exists within the folder you?re currently browsing (use ?? if you?re not sure). �?????λ???????????? ? This will take you to a speci?c location within your home folder (the ~ character tells cd to start looking in your home folder). Starting with a / will tell cd to start the path at the root folder of your hard drive. � ???? ? This ?nal useful command simply takes you up one level in the folder structure. When you get the hang of it, using a terminal as a ?le manager becomes pretty simple and quite a joyful experience. As well as cd, the three fundamental commands are mv, rm and cp. The mv command is used to move a ?le from one location to another, rm is Man up One command that?s invaluable is man, which is short for ?manual?. This will open up the help ?le for any other command. So if you want to know all the options for the ?? command, simply type ?λ??? and see what comes up. It might seem odd, if you?ve spent your life in graphical applications and utilities, but complex programs run in the text terminal, too. There are several text editors which normally come as part of the whole package, notably nano and vi. You can open a blank document by typing nano, or you can edit an existing one by typing ?λ????λ?????????????? (and do the same with vi). Some of the terminology may seem odd, though: To write out (Ctrl+O) means save, for example and so on. If you?ve been copying and pasting commands from the web all day, you might want to check what you?ve actually done. You can use history to give you a list of all the terminal commands entered going back a long, long way. Execute speci?c numbered commands with !<num>. You can go back through recent commands just by using the up and down arrows (and reissue them by tapping Enter), or search for commands by pressing Ctrl+R. 20 chmod User permissions are one of the most important parts of Linux security to understand. Every ?le has a set of permissions which de?nes who can see a ?le; who can read and write to a ?le; and who can execute a ?le as a program. A ?le which can be seen by web visitors, but can only be changed by a speci?c user, is just about as basic as it gets when it comes to locking down a server. The problem is that some ?les need to be changeable and some don?t ? think of a WordPress installation for a blog. You want WordPress to be able to write some ?les so it can update them, but there?s also a lot of ?les you don?t want it to be able to change ? and you really don?t want to give it power to execute code unless you have to. The ?ip side is that problems with web servers can be traced back to incorrect ?le permissions, when an app needs to be able to modify a ?le but has been locked out by default. Your friend in this area is chmod. It changes permissions for which users and groups can read, write or execute ?les. It?s usually followed by three digits to indicate what the owner, members of its group and everyone else can do. Each digit ranges from 0?7, where 7 allows for read, write and execute and 1 is execute only. If your user ?owns? the ?le in question, the syntax is simple. ????????????λ??, for example, will give all users the ability to read and write to a ?le. It?s good practice not to leave ?les in this state on a web server ? for obvious reasons. If you don?t own the ?le, you?ll need to add sudo to the front of that command. www.linuxuser.co.uk 39 Tutorial SimpleScreenRecorder Video capture on Linux Learn to use SimpleScreenRecorder for training purposes and make how-to videos worth their file size in gold Tam Hanna is the CEO of the Bratislava-based consulting company Tamoggemon Holding k.s. The ?rm?s focus is consulting in the development of interdisciplinary systems consisting of software, HID and hardware. Tam often makes screen recordings for his work. Resources SimpleScreen Recorder http://bit.ly/ SSRecorder Con?guration for YouTube recordings http://bit.ly/ SSR-YouTube The old-school approach to recording a walkthrough used to involve pointing a camera at your screen ? and you still see those on YouTube, especially if someone is covering BIOS/EFI guidance. While this works in theory, it is grossly inef?cient: keep in mind that your computer generates digital picture information, which is then output onto an analogue medium. An analogue camcorder then re-digitises this information, only to store it into a video ?le. In addition to the effort involved in getting the alignment right, digitisation and quantisation noise never sleep. Capturing programs used to be dif?cult because of the high processing power required for encoding live videos coming from a framebuffer. Fortunately, the development of ever-faster processors ameliorated this problem. For instance, your author often records live video of Android Studio and similarly demanding products such as PCB design software, IBF Target3001 on a ?ve-year-old IBM ThinkPad T430 with a relatively slow dual-core Intel CPU. A question of tooling Looking for screen-recording solutions in your repository of choice will yield a huge number of options. However, selecting a reliable one is not easy. For instance, we?ve experimented with Kazam and this turned out to be a disaster, as the program regularly created corrupted Figure 1 Above Manage your multiple screen setup carefully, in this case, two screens are next to one another and have a slight offset footage. SimpleScreenRecorder by Maarten Beert can be considered the gold standard and it?s used by both YouTubers and educators. The following steps to install it assume that you?re using Ubuntu 14. 04: sudo add-apt-repository ppa:maarten-baert/ simplescreenrecorder sudo apt-get update sudo apt-get install simplescreenrecorder Users on a more recent version of the operating system can download the program directly, as Canonical added the product to the repositories for 17.04: sudo apt-get update sudo apt-get install simplescreenrecorder Paper analysing frame rates used in e-learning scenarios http://bit.ly/ Screencastelearning If you use a 32-bit 3D application on a 64-bit workstation, you will need to install an additional module for recording your output: sudo apt-get install simplescreenrecorderlib:i386 Above The ?rst step is all about selecting the right data source 40 Figure 2 SimpleScreenRecorder isn?t limited to Ubuntu Linux. The program can be used on almost any *NIX operating system. For more information on support head to http://bit.ly/SSRecorder. With that out of the way, the program can be started. Simply type its name to open the main screen (shown in Figure 1). When working with SimpleScreenRecorder, start at the top and look downward. The ?rst toggle allows you to select the source of the material that you want to record. One very important aspect relates to systems which have more than one screen attached: the combo box next to ?Record the entire screen? provides a variety of options. Individual screens are identi?ed Figure 3 trade-off against using up more processor resources which are better spent on the program you are trying to demonstrate. Also, make sure to select the checkbox ?Allow frame skipping?. It is worth its weight in gold should you computer?s processor be overloaded during the recording process. Finally, make sure to set up the ?le storage properties correctly ? accidentally overwriting production-ready footage can be catastrophic. Going live Above Once you've chosen your recording settings, you get a dialog that's ready and waiting for you to press 'start recording' by the number assigned to them by Ubuntu?s display stack: if you want to record one display, you simply select its name. The selection for all screens must be managed with care. When working with multiple screens, most distros ? Ubuntu included ? try to map the display in a fashion similar to the way they are arranged on the screen. This slightly confusing concept is illustrated in Figure 2. When ordered to record all screens, SimpleScreenRecorder grabs the entire bitmap out of the framebuffer. If it contains a large amount of garbage or unwanted black space, simply rearrange the display organisation in display settings before starting the actual recording procedure. Keeping the amount of information recorded in check can be a challenge. SimpleScreenRecorder provides a number of recording options, including ?Record a ?xed rectangle? and ?Follow the cursor?. The former requires you to designate an area on the screen, whose contents are then recorded. Ergonomic experience speaks against using the second mode of operation ? the large amount of motion tends to cause nausea in viewers in a relatively short time. Two options remain on the screen: ?rst of all, you can enable the record cursor option if you want your mouse pointer to show up during the recording. This can be bene?cial when recording click-by-click instructions: if the cursor is missing, viewers have to concentrate harder and will become tired more quickly. Furthermore, the frame rate and the audio recorder can be enabled. Given that audio tends to be the biggest problem for aspiring video creators, be careful with this ? the microphones of most PCs, while perfectly suf?cient for VoIP, offer awful quality for recordings. With the initial recording options out of the way, click the ?Continue? button to switch to the second part of the setup assistant. SimplyScreenRecorder provides a large number of options for tailoring the output. We usually select an MP4 container and use the H.264 codec for encoding. Generally, set the fastest possible encoding options: using a bit more memory (in the range of less than a megabyte per second) is usually a good SimpleScreenRecorder?s GUI contains another gotcha: clicking ?Continue? opens a screen (Figure 3). This does not enable recording ? the actual recording process starts only after the ?Start recording? button is clicked. With that sorted, the actual recording can commence. The moment ?Start recording? is clicked, the program starts collecting information. Counters displayed on the screen provide an overview of the current status (Figure 4) ? if a signi?cant imbalance (more than 1fps or so) is shown between the two values, the CPU is overloaded. One very helpful feature of Simple ScreenRecorder is its native support for breaking apart Figure 4 footage. If you click ?Pause recording?, the program will commit the currently recorded information to the disk. Pressing ?Start recording? once again leads to the creation of a new recording. In addition to that, the log ?eld at the bottom of the screen provides an overview of recent events. Finally, click ?Save recording? to complete the recording process and commit the ?nal bit of information to your disk. Should you ever feel paranoid about the contents of the recording, click the ?Start preview? option. SimpleScreenRecorder will react to this by showing a small on-screen window with recently captured information: be aware that it updates much slower than the actual information, to save CPU performance. The pro?le manager displayed at the top of most SimpleScreenRecorder dialogs enables you to create and restore pro?les. With them, settings can be stored and recalled comfortably: this is very helpful in that it allows you to quickly reload commonly used settings. The developer, furthermore, provides a set of prede?ned settings for some commonly used recording scenarios. Creating good tutorial videos is an art in itself: recording perfect screencasts is but part of the problem. The best footage will not go far if it presented confusingly and unstructured way. However, obtaining great ? and noise-free ? video will simplify getting the rest right, as didactic skills can improve as you practise and audio quality can be improved with the purchase of a better microphone, such as Blue Yeti. Left The number of frames going in and out should be balanced A question of frame rate Gamers show a knee-jerk preference for high frame rates. However, when working on a tutorial, a YouTube show or something similar, needlessly cranking up the frame rate is unwise. A study performed by Springer shows that more than 50 per cent of nongame screencasts are at 15fps (see http://bit.ly/ Screencastelearning). www.linuxuser.co.uk 41 Tutorial MQTT PART ONE MQTT: An intro to the protocol powering IoT Tam Hanna is the CEO of consulting company Tamoggemon Holding k.s. He grew up under the in?uence of Eric Sink?s classic essays on coupling and event orientedprogramming. Resources https:// mosquitto.org The website of the Mosquitto server used to host our infrastructure Use MQTT and set up a Mosquitto server to send data between resource-constrained embedded devices Event-oriented architectures are ideally suited to many tasks. For example, take a system that forwards humidity sensor data to various clients. The biggest bene?t is that the arrangement between producers and consumers can be organised on the ?y: if readings must also be forwarded to a logger, no changes in the creator (the sensor) or the other consumers are required. Furthermore, testing event-driven architectures is greatly simpli?ed compared to traditional systems. This phenomenon is explicable if you look at the way in which parts are tied together ? replacing the aforementioned humidity sensor with a mock-up class is really simple: you implement the event-sourcing protocol and proceed to sending prepared data to the clients. Event-based systems are nothing new. Pivotal in the world of IoT, MQTT is a lightweight messaging protocol for small sensors and mobile devices, optimised for highlatency or unreliable networks. Its main bene?t is that it is an industry standard: developers working on a custom protocol must implement the network code on every single platform; MQTT users simply download the library. Furthermore, developers are likely to already know how to handle MQTT: this simpli?es cooperative scenarios in smart-home and similar high-value environments. Finally, there is a large variety of commercial offerings such as hosted MQTT brokers available, which means that scaling problems lose a lot of their bite. Most MQTT tutorials start with a detailed description of the protocol, only to look at practical applications afterwards. As this approach can be tedious (and we have a three-part series), let?s take a different approach. We?ll start out by setting up the Mosquitto server and will then connect all kinds of add-ons to it. Don?t install the Mosquitto server on a virtual machine: the next part of the series will deploy advanced hardware which must be on the same network as the MQTT server. Fortunately, the open source Mosquitto is available in most, if not all, package repositories. When working http://mqtt.org The website of the MQTT protocol community www.hivemq.com A professional, albeit pricey, alternative to Mosquitto Above Fun fact: MQTT was originally designed in 1999 by Andy Stanford-Clark and Arlen Nipper for monitoring oil and gas pipelines 42 on Ubuntu 14.04, the product can be deployed via sudo apt-get install mosquitto. The following steps assume that the Mosquito server is on a private network: securing it for use on a public server isn?t covered in this series. Mosquitto implements a publish-subscribe event system. This means that clients must announce their interest in a speci?c channel, and will, from then on, be provided with all events matching the string. Let?s start out by demonstrating this feature with the Mosquitto commandline clients, which are not part of the distribution. They can be downloaded by entering sudo apt-get install mosquitto-clients. MQTT?s channels are generated dynamically. Open a new terminal on your development workstation and enter mosquitto_sub -h localhost -t tamstest. The Mosquitto subscription agent will remain blocked until messages come in: the -t parameter speci?es which channel is to be observed. The distribution comes with an additional helper responsible for creating MQTT events: mosquitto_pub -h localhost -t tamstest -m "hello world" The moment you press the Return key in the terminal containing the mosquitto_pub, the ?rst terminal will show a ?hello world? message (similar to Figure 1). Bring in the Qt The MQTT module for the C++ framework is not complete as of version 5.10. Due to that, we must make do with a beta version from GitHub (https://github.com/qt/qtmqtt). As integrating the provided module into a local Qt built gets tedious, LU&D provides you with a ready-to-go project skeleton. It is based on beta 3 of Qt 5.10, and is included with the code accompanying this tutorial. It was created by copying the relevant implementation ?les into the project, and adjusting some of the includes to the new module-less situation. Load it and open the header for the MainWindow. Then, make sure to include a header and to create a pointer for the MQTT client class instance which we will use during the following steps: #include "mqtt/qmqttclient.h" . . . class MainWindow : public QmainWindow . . . private: Ui::MainWindow *ui; QMqttClient *m_client; }; With that done, let?s proceed to connecting to a server in the constructor of MainWindow: MainWindow::MainWindow(QWidget *parent) : . . . { ui->setupUi(this); m_client = new QMqttClient(this); Figure 1 m_client->setHostname("test.mosquitto. org"); m_client->setPort(1883); connect(m_client, &QMqttClient::stateChanged, this, &MainWindow::up dateLogStateChange); connect(m_client, &QMqttClient::disconnected, this, &MainWindow::br okerDisconnected); m_client->connectToHost(); } Above Here C mosquitto_sub waits for incoming messages and does not return until terminated After creating a new instance of the QMqttClient class, we set the address of the Mosquitto test server: the developer team behind the server provide a reference implementation so that developers can perform quick integration tests. Port number 1883 is considered standard for unencrypted MQTT ? enabling encryption features is not covered in this guide. The MQTT module for the C++ framework is not complete as of v5.10? we must make do with a beta version from GitHub Finally, two event listeners are registered so that we can collect information about the connection progress. They output messages into the qDebug log ? updateLogStateChange looks like this: void MainWindow::updateLogStateChange() { const QString content = QDateTime::currentDateTime().toString() + QLatin1String(": State Change") + QString::number(m_ client->state()); qDebug() << content; } Connection status information is given in the form of a C enum ? be aware that a successful connection leads to the value 2 being stored in the state ?eld. With that accomplished, it?s time to test our program for the ?rst time. The output of StateChange will inform us that the test service has accepted our connection ? in theory, we can now use our own server Beware of licensing! As of this writing, the Qt company has not completed the licensing procedures on QtMqtt. Be careful when including the code in a commercial application which is not GPL licensed and expect ? usually minor ? API changes as the product achieves technical maturity. www.linuxuser.co.uk 43 Tutorial Dynamic logging Mosquitto is not limited to outputting debugging information to the syslog. It can also make status information available via a topic, which random MQTT clients can register in order to stay in the loop. Further information on this ? very advanced ? use case can be found by visiting http://bit.ly/ SYSTopics. MQTT by adjusting the connection parameters passed into the MQTTClient object: MainWindow::MainWindow(QWidget *parent) : . . . m_client->setHostname("localhost"); m_client->setPort(1883); Sadly, running this modi?ed version is unsatisfactory. After outputting ?State Change 1? to tell us the connection process is started, a broker-lost message is emitted. The server is actively refusing our connection requests. Hunting down trouble Mosquitto is supported by a wide array of logging facilities. Sadly, most of them are not enabled by default Firing them up requires editing of the ?le /etc/mosquitto/mosquitto.conf ? pick an editor of choice, start it with superuser rights and get to work. All important parameters are found in the segment logging ? the logging destination must be set up by passing in the string syslog: log_dest syslog Next, look for the following lines. They are pre?xed with a # character by default, which marks them as disabled. Simply remove the character to enable the relevant logging features and then save the ?le: log_type error log_type warning log_type notice log_type information connection_messages true MQTT protocol received an update which is implemented in both QtMqtt and the of?cial Mosquitto test server. However, the team behind the Ubuntu package manager is known for being slow to update things. Due to that, a bit of manual intervention is needed. First of all, return to the syslog ? a starting instance of Mosquitto will announce its version. Be careful to ignore the build date ? it is relevant only in that it describes when Ubuntu?s build server last touched the code, and has nothing to do with the freshness of the code base used. Next, remove the existing version of both client and server from your workstation by entering sudo aptget remove mosquitto mosquitto-clients. Finally, add a new PPA to apt-get?s list, perform update and reinstall Mosquitto: sudo add-apt-repository ppa:mosquitto-dev/ mosquitto-ppa sudo apt-get update sudo apt-get install mosquitto During the deployment of the new package, a message will pop up. Apt-get asks if you want to use the current con?guration ?le or prefer to replace it with the default one provided in the new version of the package. As we want to keep the logging commands intact, select the N option and press Enter to keep the existing con?guration ?le. After that, that system log ?le will show a new version of the server start up automatically: its build date might be older than the one obtained from the of?cial repository; its version number will, however, be in the range of 1.4.x, indicating a much newer version of the underlying code base (see Figure 3). A look at channel matching Below Our Qt program sends something which our server does not accept Restart the Mosquitto server to force it to read the new con?guration ?le by entering sudo service mosquitto restart. Looking at logging info contained in the system log is best accomplished using the syslog viewer ? enter ?System Log? and feast your eyes on the contents. Then, reconnect your client to get the error shown in Figure 2. This problem is caused by a versioning mismatch between Ubuntu and QtMQTT?s implementations: the Figure 2 Let us modify our Qt program so that it registers itself for our test message. First of all, the creation of additional ?elds in the header of the main form is required: #include "mqtt/qmqttsubscription.h" class MainWindow : public QmainWindow . . . QMqttClient *m_client; QMqttSubscription *mySub1, *mySub2, *mySub3; }; QtMQTT describes subscription relationships via the QMqttSubscription class. We create a total of three instances here ? this makes adding further subscriptions easier when you want to handle more than one of them. At this point, a signi?cant question remains: where do we start the actual subscription process? The answer to this can be found in updateLogStateChange: void MainWindow::updateLogStateChange() { . . . if(m_client->state()==QMqttClient::ClientS tate::Connected) { 44 mySub1=m_client>subscribe("tamstest",0); connect(mySub1, &QMqttSubscription:m essageReceived, this,&MainWindow::updateMessage); connect(mySub1, &QMqttSubscription:: stateChanged, this, &MainWindow::updateStatus); } } After outputting information into the debug console, the event handler checks the connection state of the MQTT client. If a connection to the broker has been established successfully, the subscribe() method is invoked in order to spawn a subscription class. Its constructor requires both a string describing the channel and a numeric QoS ?eld ? the letter will be discussed in next issue?s tutorial. We ?nally install two signal handlers in order to make sure that our program is noti?ed of signi?cant events: void MainWindow::updateMessage(const QMqttMessage &msg) { qDebug() << "Got " << msg.payload(); } void MainWindow::updateStatus(QMqttSubscriptio n::SubscriptionState state) { qDebug("State Change!"); } With that done, restart the program. Enter the mosquitto_pub -h localhost -t tamstest -m "hello world" command to ?re off another message. It will show up in the debug log. Figure 3 >subscribe("tamoggemon/labvac/lecroy1",0); connect(mySub1, &QMqttSubscription:: messageReceived, this, &MainWindow::updateMessage); Above Mosquitto 1.4 fully implements more recent MQTT standards Dispatching an event simulating this unlucky oscilloscope is accomplished via mosquitto_pub -h localhost -t tamoggemon/labvac/lecroy1 -m "hello world". The interesting aspect of topics is the use of wildcards: it allows programs to sign up for a group of events with a single operation. We can, for example, subscribe to all events emanating from all devices based in Vac: if(m_client->state()==QMqttClient::ClientS tate::Connected) { mySub1=m_client>subscribe("tamoggemon/labvac/#",0); One of the design goals of MQTT is the ability to handle a very large number of events Complex games! Using strings to correlate between event types works as long as the number of events is small. One of the design goals of MQTT is the ability to handle a very large number of events. For this, some kind of structure is required: for example, feeds could be structured to have parents and children. This is accomplished by the creation of a folder hierarchy: the ?/? character is used to keep the individual elements apart. Let us look at the following structure, which could represent a network of devices inside of two laboratories owned by Tamoggemon Group: tamoggemon/labbratislava/dpo1 tamoggemon/labbratislava/lecroy1 tamoggemon/labvac/lecroy1 tamoggemon/labvac/lecroy2 From Qt?s point of view, subscribing to such an event is not particularly dif?cult. Let?s subscribe to events from the ?rst LeCroy oscilloscope in Vac ? the main issue is that the strings are case sensitive: if(m_client->state()==QMqttClient::ClientS tate::Connected) { mySub1=m_client- Alternatively, we can disable one layer of the hierarchy. For example, let?s assume that you are interested in all the events generated by an oscilloscope that we've called lecroy1: if(m_client->state()==QMqttClient::ClientS tate::Connected) { mySub1=m_client ->subscribe("tamoggemon/+/lecroy1",0); While this is quite ?exible, keep in mind that MQTT does not allow for complex string matching. For example, it is not allowed to use matchers such as lecroy* to announce interest in all LeCroy units. Furthermore, changing the structure of an MQTT deployment tends to be extremely dif?cult: due to this, developers are very advised to think carefully about how they structure their product before the ?rst piece of hardware gets sent out into the ?eld. Running MQTT on the desktop is child?s play: packet loss is unlikely. This is not the case in the real world. MQTT contains a variety of features intended to mitigate or alleviate the consequences of connection losses. In the next issue, we will deploy MQTT on Android Things. Until then, may your packages always arrive safely. www.linuxuser.co.uk 45 Tutorial Computer security Redirect and intercept HTTP requests from devices Toni Castillo Girona holds a degree in Software Engineering and an MSc in Computer Security and works as an ICT research support expert in a public university in Catalonia (Spain). Read his blog at http://disbauxes. upc.es. Resources Burp Suite Free Edition http://bit.ly/ BurpSuite By redirecting traffic from phones and IoT devices, you can detect malicious behaviour, find weak spots and more Proxies are great tools for pen-testing web applications. They are capable of detecting common vulnerabilities on websites (SQL-i, XSS, sensitive data exposure, and so on), while providing you with a powerful framework for circumventing website client-controls, performing manually scans or conducting automated attacks. A lot has been written about using either Burp or ZAP! proxies during a web pen-testing engagement. But it is not all about the web ? think about all those embedded and IoT devices making use of the HTTP protocol behind the scenes in order to send or receive data, communicate with some server in the cloud, or just perform some other unattended tasks. Not to mention smartphones running either Android or iOS! Most of the apps you are using on a day-to-day basis employ the HTTP protocol too; using a proxy can help you identify those weak spots or even detect some hidden functionality or malicious behaviour. You can redirect traf?c from your smartphone or IoT devices as easily as with any other computer and then feed it to Burp or ZAP! This way you will be able to intercept HTTP requests, manipulate them, and perform the usual sort of analysis and attacks as with any regular website. Redirect (some) traffic When it comes to redirecting, monitoring or even modifying HTTP requests from your devices, you will face two situations: either you have administrator privileges on the device or you don?t. The former will allow you to con?gure the device network with your own under-yourcontrol gateway or proxy (in case the device supports proxies, that is). The latter will leave you with two options: Wireshark apt-get install wireshark Arpspoof from the Dsniff package apt-get install dsniff NoPE Proxy http://bit.ly/ NopeProxy A GNU/Linux computer An Android device Above Burp intercepting an Android app. This is the virtual equivalent of looking through a peephole 46 either you can redirect traf?c by setting up some rules on your network router (assuming you do have administrator rights on the network router, of course) or you poison the device?s ARP cache with an ?ARP spoo?ng attack?. Let?s focus ?rst on devices for which you have admin access. Most IoT devices that use Wi-Fi functionality to communicate with the network can be automatically con?gured by means of the DHCP service. Some others may allow you to set up their TCP/IP settings manually. Android devices can be con?gured to use a proxy server as well: follow the next steps in order to intercept any HTTP request from your smartphone using Burp (bear in mind that these steps may vary depending on your Android version and device vendor): 1. On your GNU/Linux computer, execute Burp and go to the Proxy/Options tab. 2. Push the ?Add? button below ?Proxy Listeners?. Type 8080 in the ?Bind to port? text box and then select your private IP in the drop-down list ?Speci?c address? (it should be something like 192.168.X.X). Finally, press the ?OK? button. A new proxy listener will be shown. 3. Make sure to disable traf?c interception by pushing the ?Intercept is ON? button on the Intercept tab. Then go to the HTTP History tab. 4. On your Android device, go to Settings > Wi-Fi. Longpress your Wi-Fi ESSID and select ?Modify network?. 5. Enable the checkbox ?Advanced options? and scroll down a bit until you ?nd the Proxy drop-down list widget. 6. Tap it and select ?Manual?. 7. Type your computer?s IP address in the ?Proxy hostname? text box, 8080 in the ?Proxy port? text box and then tap ?Save?. Now open your favourite web browser on your smartphone (e.g. Google Chrome) and browse to any nonSSL website. You will see some activity on the ?Proxy/ HTTP history? tab in Burp. As clearly stated by Android: ?The HTTP proxy()? may not be used by the other apps?. You can redirect traf?c from your smartphone or IoT devices as easily as with any other computer Try it: toy with some apps that make use of the internet, such as Ted Talks and the Merriam-Webster dictionary; for some you will see HTTP requests going through Burp whereas for some others you won?t. If you try to connect to TLS-protected websites, however, you will get an error message because of a certi?cate mismatch. You can ?x that by installing a Burp SSL certi?cate on your device: 1. In Burp, go to the Proxy/Options tab and click the ?Import/Export CA certi?cate? button. 2. Select ?Export/Certi?cate in DER format? and then click ?Next?. 3. Save the ?le somewhere in your computer; name it as burp.crt. 4. Now, upload this ?le to your Android device (e.g. by means of the adb command). 5. On your Android device, tap Settings > Security > ?Install from (phone) storage?. Choose burp.crt as the new certi?cate to install. Give it a name (e.g. BURP) and then tap ?OK?. 6. If you tap Settings > Security > Trusted credentials/user, you will see the new installed certi?cate. Tap it to get some information (such as its ?ngerprint and validity). If you navigate to TLS-protected websites this time, no security warnings will be shown at all. Besides, the HTTP History tab in Burp will show you all the TLS traf?c in cleartext (i.e. decrypted). This is so because Burp is now performing a proper ?man-in-themiddle? (MITM) attack. Make sure to remove this certi?cate once you are done with this tutorial. Attackers who might be able to get their hands on its private key would decrypt your TLS connections just as easily. Above How miserable: you?ve just managed to perform a MITM attack against yourself! Redirect (all) traffic So far so good but? what about those apps that do not use the proxy? Not to mention those IoT devices that do not have an option to set up a proxy in the ?rst place! Enter iptables and IP forwarding! Open a new terminal on your GNU/Linux computer and enable IP forwarding ?rst: # echo 1 > /proc/sys/net/ipv4/ip_forward Get back to your Android device and note its current IP address: Settings > Wi-Fi > Advanced. You are going to set up your computer as a router for your smartphone by using the iptables FORWARD chain. In order to forward any packet from/to your device, type the following commands (replace <ethX> with the network device that is connected to the same network as your Android device and <IP_ANDROID> with your phone?s IP address): # iptables -t nat -I POSTROUTING 1 -o <ethX> -j MASQUERADE # iptables -I FORWARD 1 -o <ethX> -s <IP_ ANDROID> -j ACCEPT # iptables -I FORWARD 2 -i <ethX> -d <IP_ ANDROID> -j ACCEPT Smart ?rewalls With all the threats IoT devices have to face, some vendors have started to develop a new generation of smart ?rewalls. These devices monitor the network and ? thanks to their cloud infrastructure, big data and deep learning techniques ? they can spot suspicious activity, send warnings or even take a proactive approach by blocking packets. Check out these examples: Cujo, Norton Core, RATtrap and Bitdefender Box. Please note that if you have the default policy of the www.linuxuser.co.uk 47 Tutorial Burp and SSLStrip-like attacks For those devices that are using TLS to protect their communications, you will not be able to decrypt their packets by just ARPpoisoning their caches. SSLstrip to the rescue! Surprisingly enough, and as long as the HSTS header is not present and the ?rst connection is through HTTP, this attack still delivers. Burp includes some options in order to perform SSLstriplike attacks, too (see http://bit.ly/ SSLstrip). Tutorial ?les available: ?lesilo.co.uk Below Maybe you don?t see them right away, but ads are always a few HTTP GETs away! 48 Computer security FORWARD chain set to ?ACCEPT?, you don?t even need to add the last two rules at all. We encourage you to set your default ?rewall policy to DROP always, though. Now your computer is ready to start forwarding packets from/to your device. Next, set up the gateway manually on your smartphone: 1. Go to Settings > Wi-Fi, long-press your Wi-Fi ESSID and select ?Modify network?. 2. Enable ?Advanced options? and tap ?IPv4 settings?. Select ?Static?, then type the original IP address of this device in the ?IPv4 address? text box and your computer?s IP address in the Gateway text-box. 3. Tap ?Proxy? and select ?None? to disable the proxy. Finally, tap ?Save?. You can open any app and use it the normal way: this time, however, instead of using your router/AP in order to access the internet, the entire system will be routing all the network traf?c through your computer. You can start capturing some traf?c right away with Wireshark: execute it and select the network adaptor that has the IP address you have set up on your Android device as its gateway. In order to capture only packets coming from or going to your smartphone, set this capture ?lter: ?host <IP_ANDROID>? and disable the ?promiscuous mode?. Start using some apps (such as Telegram and your web browser) and you?ll see a lot of packets being captured by Wireshark. All TLS packets will be encrypted, though. This is because you are not forwarding these packets to Burp proxy. Therefore, they cannot be decrypted because your computer is merely forwarding packets from/to your device dumbly and capturing them along the way. So the next step is to forward some of these packets transparently to Burp. Let?s imagine you want to see every packet destined to or received from ports 443/TCP and 80/TCP. On your GNU/Linux computer, type this in the terminal: # iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8080 # iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080 Execute Burp and make sure it is listening on port 8080/TCP. Because now you want to redirect every HTTP request to ports 443 and 80 transparently, you have to set up Burp proxy in ?transparent? mode. Otherwise apps won?t be able to connect to the internet because they are not aware of the proxy at all (this sort of proxy is known as a ?Transparent Proxy?): 1. In Burp, go to the Proxy/Options tab. 2. Select the proxy listener you have con?gured previously and click ?Edit?. 3. Go to the Request Handling tab and enable the option ?Support invisible proxying()??. Finally, hit the ?Save? button. Get back to your Android device and make sure the Burp certi?cate is installed. Then, execute some apps that make use of TLS encryption (e.g. LinkedIn). The HTTP History tab in Burp will show you every single HTTP The HTTP History tab in Burp will show you all the TLS traf?c in cleartext, i.e. decrypted request and response in cleartext. Some apps may check the server certi?cate; for these apps the connection will irremediably fail. Of course, there may be apps that use different ports for their HTTP requests; by using the PREROUTING chain in the NAT table, you will be able to feed Burp anyway. If you do not know beforehand which port a particular app is using, open Wireshark and capture some traf?c. After a while, you will have some packets from your device sent to an external IP address on a certain port. Then you can add a new rule to the PREROUTING chain for that particular port. Forwarding packets allows you to detect unusual patterns on your devices, too (what follows is the cheap way; for some advanced protection, see Smart Firewalls box, p47). Imagine that you are monitoring all the packets coming from your new gadget (a smart light-bulb). When the device is working normally, you see some HTTP activity encapsulating some JSON data coming back and forth between your smart light-bulb?s IP address and a remote server in the cloud (e.g. Amazon S3). This is its normal behaviour, so to speak. So you set a capture ?lter such as: src host <LIGHTBULB_IP> and leave Wireshark capturing data for a while. Later on, you use the Statistics > IPv4 Statistics > ?Source and Destination Addresses? menu option and you ?nd out that, besides the remote Amazon S3 server you consider legit, there is another IP address you haven?t seen before. Its count column (number of packets sent to this destination address) is worryingly high. You navigate to Robtex (www.robtex.com) and paste this IP address in the search text box. Robtex tells you that this an IP from Russia. You do your research and, apparently, this remote IP has nothing to do neither with your vendor?s light-bulb nor with the cloud service it uses. The domain it belongs to is reportedly malicious. Indeed, your smart gadget has been p0wned! Congrats: now it is a new zombie working for the zillionth botnet! Perform MITM attacks How about those devices you cannot control? Let?s imagine the device is connected to a network you don?t manage; setting up some rules on the network router is out of the question, too. Next thing could be to set up your own rogue DHCP server, but maybe the device?s TCP/IP settings have been con?gured as static. So what?s next? Performing a classic MITM attack by spoo?ng its ARP cache surely won?t hurt. This way all its traf?c will be redirected to your computer and you will become its gateway instead of the actual one on the network. Sometimes this won?t work at all or it may work partially. On some occasions, the gateway may have a static ARP entry for particular devices (the other way round is frankly unlikely). So let?s imagine you want to impersonate the gateway (192.168.1.1) in order to capture all the traf?c coming from your device ?DEVICE?. Execute the arpspoof command this way: # arpspoof -i <ethX> -t <IP_ DEVICE> 192.168.1.1 2> /dev/null& # arpspoof -i <ethX> -t 192.168.1.1 <IP_DEVICE> 2>/dev/null& Now you can run Wireshark and you will see all the packets from DEVICE. Of course you can redirect some of these packets to Burp as well. Do it now: make sure to add the desired rules to redirect any non-SSL HTTP And don?t forget to start Burp on that computer too, listening on port 8080/TCP. Once you are done with the poisoning, kill both arpspoof processes by executing killall arpspoof. # iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --todestination <ANOTHER_IP>:8080 Alternatives to Burp Intercept non-HTTP traffic Burp deals with HTTP and HTTPS requests. But of course, some devices may be using their own sort of protocol to communicate. Josh Summit has developed a Burp extension that allows non-HTTP interception within Burp: NoPE Proxy (http://bit.ly/NopeProxy). Install telnetd on your GNU/Linux now: apt-get install telnetd. Then, download and install NoPE Proxy (see Resources, p46 ). Open Burp and go to the Extender tab. Push the ?Add? button and then ?Select File? besides ?Extension File (.jar)? and select NopeProxy.jar. Then click ?Next?. Once the extension is successfully installed, go to the NoPE Proxy/Server Settings tab. You are about to intercept Telnet packets so, under ?Not HTTP Proxy settings?, type your computer?s IP address in ?Server Address?, 23 in ?Server Port?, 9999 in ?Listening Port? and leave ?Certi?cate Server? empty. Finally, push the big green ?+? button to add this new listener to the list below. Make sure to enable this listener by clicking on the ?Enable? checkbox. Now open a new terminal and redirect port 23/TCP to this listener: # iptables -t nat -A PREROUTING -p tcp ?dport 23 -j REDIRECT ?to-port 9999 Next, install a Telnet client on your smartphone and make sure that the MITM attack is still going on against your device. Then, try to connect to your computer using Telnet. You will see some packets on the TCP History tab in Burp. Once you are presented with the ?login? prompt, and before typing any username, push the ?Intercept is OFF? button on the TCP Intercept tab. On your device, type any login and send it to the server. You will see a new Telnet packet on the TCP Intercept tab waiting to be manipulated before being sent for real. Change the login string to something else by editing the text right in this tab and then disable ?Intercept is ON?. Finally, forward the packet by pushing the ?>>? button. You will see how the server is now expecting a password for this new login! Forwarding packets allows you to detect unusual patterns on your devices, too request to Burp as seen before. If you want to redirect the traf?c to some other computer under your control, that is possible too. Use DNAT instead of REDIRECT: WHAT NEXT? 1 OWASP Zap! Proxy The Zed Attack Proxy (ZAP) from OWASP is an incredible and powerful open source proxy quite similar in functionality to Burp. It ships with a lot of automated tools to scan for vulnerabilities in web applications and it can be automated thanks to its powerful REST API. http://bit.ly/OWASP-ZAP 2 mitmproxy This is an open source MITM proxy that allows HTTP and HTTPS interception with an ncurseslike graphical interface. If you are a fan of tcpdump, try mitmdump instead. You can install it right away on your Debian-based distro: apt-get install mitmproxy. https://mitmproxy.org 3 TamperData An outdated Firefox add-on, but still functional. It is as simple as it gets: it can intercept and manipulate POST parameters. It is quite unstable, but perfect for quick-and-dirty tests on-the-?y. http://bit.ly/TamperData 4 Charles Proxy Another MITM HTTP and HTTPS proxy that includes some interesting functionalities such as bandwidth throttling to simulate slower internet connections, a full AJAX-debugger engine and support for AMF 0 and AMF 3 parsing. www.charlesproxy.com/download 5 Telerik Fiddler A MITM proxy too (it can decrypt HTTPS requests), it has been designed to work on Windows OSES although thanks to the Mono framework it can also work on GNU/Linux. It also supports extensions developed using any NET language. www.telerik.com/?ddler www.linuxuser.co.uk 49 Tutorial Arduino Use Arduino?s sleep mode to monitor temperature Alexander Smith is a computational physicist. He teaches Arduino to grad students and discourages people from doing lab work manually. Resources Arduino / Raspberry Pi / Computer to receive Arduino Nano (or barebones) to monitor Temperature Sensor: DHT22 Protect your pipes and heat your home effectively by monitoring the temperature over the season Temperatures are beginning to drop to freezing and the days are already short. For those worried about spending all their money on running the boiler, January is often the most agitating month. You?ve already bled your radiators and are resigned to wearing jumpers, but are still concerned about the pipes freezing and bursting. In this issue of LU&D, we?ll be creating an Arduino temperature monitor which can be placed anywhere in the house and run for months on end, automatically reporting its data back to base so that you can make informed decisions about when to turn the heating on. We?ll begin by taking a lower-power Arduino and preparing the temperature sensor. In LU&D183 we used an LM35, so in this tutorial we?ll show you how to interface with the DHT22 using a library provided by Adafruit. We?ll then add the radio transmitter to send messages containing the measurements, and create a base station to receive the data. Finally, we?ll utilise the Atmel ATmega?s sleep mode, waking up every so often and powering the board only when required so that the battery doesn?t drain all at once. In the end, we?ll produce a temperature monitor which can run for the whole season without any maintenance, and gather incredibly rich data about temperature cycle of the coldest room in your home. From this you should be able to tell if you really should be turning your radiators turn on (or hopefully off) a few hours earlier. For this tutorial, your choice of Arduino matters substantially. A battery only has so much capacity, measured in milliamp hours (mAh). Most small power banks deliver 5V with a capacity of around 2200mAh. An Arduino Uno will use about 45mAh in idle mode, which equates to about 48 hours of operation before it powers down. This consumption is due to the large number of hardware components sitting on the board, all of which require power even if they are not being actively used. To reduce this burden on a battery, it?s a good idea to make your own Arduino from parts and with as few Radio transmitter & receiver (or Bluetooth, Wi-Fi adaptor for Arduino) Battery power (a power bank, coin batteries or AA batteries and case) Adrafruit Sensor library http://bit.ly/ AdafruitSensor Virtual Wire library http://bit.ly/ VirtualWire Python www.python.org Above Arduino Nano, with temperature sensor, transmitter and coin battery 50 components as is required. These ?bare-bones? boards can eliminate two-thirds of the power wastage and open the possibility of operating the microcontroller in microamps of current instead ? potentially allowing operation for years. If this is beyond your skill set, we recommend using a smaller board, such as the Arduino Pro which ships with less hardware. However, for this demonstration, we?re going to be using the Arduino Nano and it should be able to run for a few weeks before needing a change of battery. Prepare the sensor The DHT22 is a temperature and humidity sensor, available cheaply online. It comes in two variants, one with a circuit board attached and one without. If you purchase the version without the pre-built circuit, you will need to add a pull-up resistor between the voltage supply and the data pins, as communication is performed by creating a drop in voltage between data and ground. The data sheet suggests a 1k resistor, but it?s not uncommon to see a 10k resistor online. Connect voltage-in, ground and data to the digital pins on your Arduino, but avoid the pulse-width modulated pins ? denoted by a tilde (~). To interface with the sensor, we are going to use the Adafruit Sensor library and DHT Sensor library. Both are available for download through the Arduino IDE, which also handles the installation. This saves us from having to write our own program to deal with the communication protocol between the microcontroller and the DHT22. Brie?y, the Arduino sends a drop in voltage to the sensor to say it?s waiting for data. The sensor then con?rms it received the instruction and then sends a stream of 8 bits for each reading. The Adafruit library handles all this for us and deals with the pulse timings. It?s time to start programming! Open the Arduino IDE and create a new sketch. To set up the temperature sensor, we need to include a few lines at the top of the code so that the library knows how to handle the sensor and which pin to use for communication. Make sure DHTPIN is set to the data pin connected to your Arduino. We then create a DHT object which provides methods that handle the communication and conversion of data: #include "DHT.h" ??????'M¶�Rr? ??????'M¶¶Ϋ�-'M¶?? 'M¶????'M¶�Rr?'M¶¶Ϋ�-?? In ?setup?, we then need to call ??????????? to initialise the DHT object and then set the digital pins to 5V. Then only one method is needed to get a reading for temperature and another for humidity which can be performed in the loop function: ??λ???????λ???????????λ?¶?????λ??????? ??λ????????????????λ?M?????????? The sensor will take a couple of seconds to acquire a measurement and also needs two seconds to ?warm Above DHT22 ? sensor for measuring temperature and humidity up? after supplying power before it can be used. This point will become important later when we begin to turn hardware on and off to save power. You should now be able to measure the temperature in your home. If you?ve got a thermometer, you might want to con?rm if this reading is accurate; if not, you might want to add a calibration constant to correct your Arduino reading. Add the radio transmitter We now have an Arduino temperature monitor set up. For it to be useful, we?re going to need to consider how we will access the data. Whilst an SD card adaptor might be an easier option, it would interrupt the measurements to have to keep removing and inserting the SD card periodically. In this tutorial we are, instead, going to The Adafruit library saves us from having to write a program to deal with the communication protocol broadcast our data using an AM radio to a receiver connected to another Arduino which will relay that data to computer over USB. If you have a Raspberry Pi, you could connect the receiver to that and skip the second Arduino entirely. We?ve used the MX-FS-03V and MX-05V ? a 434MHz transmitter and receiver pair that can be purchased online for pennies. With two separate pieces of spare wire soldered onto the transmitter and receiver and operating at 5V, communication was achieved at distances of 50m (not in line of sight). These should do the trick at short ranges within the home. If your walls are simply too thick and range becomes an issue, you may have to switch to Bluetooth or Wi-Fi instead. Tutorial ?les available: ?lesilo.co.uk No wheel reinvention With Arduino there?s often no need to go around trying to write a sketch to handle the communication protocol between two devices. There are dozens of libraries available for installation through the IDE for doing the tricky stuff. You can then spend your time trying to make a more useful device. www.linuxuser.co.uk 51 Tutorial Every little helps? For those of you determined to get every second out of your battery, someone has performed an investigation into how effective each of these powersaving measures is. Further info is given at http://bit.ly/ GammonPower, including techniques such as reducing the internal clock speed and running your Arduino at voltages below 5V, providing some example sketches. Arduino Connect the transmitter unit to the Arduino. You should be able to power the unit from the digital output pins as with the DHT22. Connect the data pin to a digital pin, too. To interface with the radio, we?re going to be using the VirtualWire library, which is available online but may be built-in on some versions of the IDE. VirtualWire is another wrapper library which handles the communication for many radio modules. All you have to do is designate the data pin and provide a message to send across, stored in a character array. To set up the radio transmitter, you?ll need to set the high and low pins as before, initialise the library with the correct pin, and pick a bit rate. You may also need to invert the signal for some radio modules ? the easiest way to ?nd out is to play around. ?????????????????¶Ϊ'λ?λ?? ?????????????????????????? ??????????????? You now need to form a message to send to the receiver. The plan is to take a temperature reading from the DHT22 (a ?oating-point variable) and convert it into a character array. This can be done easily on the Arduino using the double to string function: ?????????. VirtualWire will then take the character array and transmit it. This can all be done in a few lines in the main loop: ??λ? ????λ??????? ?????????λ?λ¶?©???? ?? ?? ????λ???? ???????????? ????????????? ????λ???????? Receiving messages on an Arduino is a very similar process. In setup, the reference to tx (transmit) becomes ?? (receive) and in loop, send becomes ????????λ??. The received message is then stored in a buffer of unsigned integers. In a separate sketch, make a program that will attempt to receive data over radio inde?nitely and upload it to a second Arduino. You can connect the receiver in the same way as the transmitter and let VirtualWire know which digital pin is for data. This process should be fairly straightforward, but both sketches have been uploaded to the cover disc. You should be able to iterate over the message buffer letter by letter, converting from integer to character, and print the buffer to serial: The easiest way to log the data from the Arduino is probably using a Python script. In the previous tutorial (LU&D 185) we covered sending messages over serial using Python from a Raspberry Pi. We?re going to do much the same thing, but backwards. Then we can save it to ?le, with the program running continuously in the background just waiting for a message to come through. Open a new Python script. To begin with you?re only going to need to import the ?serial? library so that we can interface with the Arduino. In the Arduino IDE you should be able to view the name of the port that the Arduino is connected to under Tools > Port. In the Python script, you can then write: λ?????? ? ????λ??©???λ??????rλ??? ?to initialise the connection with the Arduino. We then simply open a new text ?le with the ?????? command and, until we exit the program, can read in characters from the serial buffer and write them to that ?le. ?λ?λE??? ? ???????r??E??? "??"? ????? ¶???? ? ? λ?????????λ???????????"?????"? ?λ?λE???????????? You may want to reformat the incoming data to make it easier to read. With the ?datetime? library, it?s easy to create a string containing the current system time, which can then be added to the end of each line in your text ?le. The easiest way to log the data from the Arduino is probably by using a Python script After running both of your Arduinos and collecting data on your computing by running the script, you should have accumulated some interesting data about the temperature in your home. With Python?s matplotlib library (or your favourite graphing program), you should ?? ????????????λ??????? ???????? ? ??? ???? ? ? ?? ? ? ?????? ???? ? ??λ? ? ? ??????? ©???λ??????????? } } Log the data Now we have received the message, we can relay it to a computer so that we can store the data and analyse it day-by-day rather than just collecting it from an SD card after a few weeks or months. 52 Above Radio transmitter and receiver pair; we've soldered a piece of wire to each to improve the range be able to plot a scatter graph of your temperature data against time. Read the data sheet By this point you?ve already made something incredibly useful. However, we?re going to take it one step further and make your Arduino as energy-ef?cient as possible. The bene?ts are obvious: by using the microcontroller chip?s low-power mode and turning off pieces of hardware, we can not only save a few pennies in electricity, but also run the device off of a battery for much longer. With the right setup, it is possible to create something which can monitor and report the temperature that can run for months without you needing to interfere. The following should work for the ATmega328 and 328P, which is available on many Arduino boards. You should familiarise yourself with the data sheet for your chip before proceeding ? it is important that you are aware of how the Arduino can be ?woken up? after it is ?put to sleep? and how this affects your ability to interface with the board. To begin, you will need to include three header ?les which let you control the chip?s hardware: ?????????λ??????????? ?????????λ??????????? ?????????λ????????? These allow you to put the microcontroller chip into one of the many sleep modes, turn off the hardware and interface with the watchdog timer ? which we?ll use to turn the Arduino back on again. To begin, start by turning off the wasteful hardware. You won?t need items such as the analogue-to-digital converter, the counters, the serial interface, even the USB itself. In loop, after interfacing with the sensor and the radio, turn off the wasteful hardware by writing: ' ©΅??? ??????λ??????λ?????? Set an alarm and go to sleep For the Arduino to wake up, we?ll need to set the watchdog timer ? one of the few pieces of hardware that doesn?t turn off. This interrupts the Arduino after a set number of clock cycles. First we allow changes to be written to the watchdog register and then, referring to the data sheet, we set the length of time the watchdog should wait before waking the Arduino (in this case eight seconds), we then reset the timer to zero: Υ'¶ ©΅?????Υ' -??????Υ'-?? Υ'¶ ©΅?????Υ'R-??????Υ'�?????? ?Υ'�??? ???????????? We?ve now cut off the hardware and asked for a wakeup-call. It?s time to send the Arduino to sleep. First we select the sleep settings: ?SLEEP_MODE_PWR_DOWN? is the deepest sleep mode, with the largest savings. Sleep Above The ?nal device, assembled and placed in an old jar mode is subsequently armed, executed and disarmed after waking. ???????????????©f--�?p{'-?�Υ΅?'{Υr?? ????????λ?????? ???????????? ?????????λ?????? At this point you won?t be able to do much to the Arduino. You?ll just have to wait for it to wake up again. It?s worth adding a large delay while testing, in case you need to upload something to the Arduino. Expect to spend a dozen minutes or so pressing the reset button and trying to ?nd a window in which you can upload a new sketch before it falls asleep again. Be careful here. Upon waking, the ATmega will execute a system reset unless an alternative function is de?ned. You can do this in a few lines at the top of the code. We?ll just use it to turn off the watchdog timer, so that the rest of our sketch runs nicely. The Arduino will then carry on where it left off before going to sleep. R©΅?Υ'¶??????? ???????λ?????? } A line-by-line explanation of how to initiate power saving is available in the example sketch provided on the cover disc. It also comes with a few extra safety measures. A full description of the energy savings is available at www.gammon.com.au/power. These measures should allow you to get the most life out any battery powering your Arduino project. If you?ve gone as far as to use a ?bare-bones? board, it?s possible to get years of operation out of a relatively small battery. Having followed the tutorial this far, you?ll now have a pair of devices which are able to measure the temperature, broadcast that over radio and store the data over the coming winter. Now, without guilt, feel free to turn your radiators on. www.linuxuser.co.uk 53 Tutorial Java PART FIVE Making a game in Java: Dependency injection John Gowers is a university tutor in Programming and Computer Science, with a strong focus on Java. He likes to install Linux on every device he can get his hands on. The Spring Framework is one of the most popular Java tools, and this project is a great opportunity to learn more about it Resources OpenJDK 1.8 See your package manager or download from openjdk.java.net JavaFX 8 See your package manager or download from openjdk.java.net/ projects/openjfx Eclipse IDE See your package manager or download from eclipse.org Spring Framework Installation instructions in article 54 This month?s Java tutorial has a slightly different focus from previous issues. Rather than make any substantive modi?cations to the functionality of our code, we will be rewiring our existing code in order to ?t it into a Java software framework called Spring. Unlike traditional programming, where we take responsibility for creating an entire program, when we work with a software framework, our focus is on providing components for the framework and con?guring it to use them. The way we have set up our program, with important classes backed by interfaces, makes it well suited to this style of programming. We will be using the Spring Framework, which is probably the most popular framework for Java. Using Spring will allow us to incorporate an important design pattern ? dependency injection ? which cannot be used with Java alone. If you haven?t been following the series so far, you can get up to speed with a working version of the assignment from last month. Find the ?le eggs.tar on the cover disc, and download it to your system. In Eclipse, select General > Existing Projects into Workspace, and then click ?Select archive ?le?, followed by ?Browse...?. Navigate to eggs.tar and click ?Finish? to import the project. Dependency injection The key concept with dependency injection is that of a dependency between classes. A class Master has a dependency on a class Service if the class Service is mentioned somewhere in the code of the class Master: for example, if Master has a ?eld of type Service. Rather than have an explicit dependency between the two classes, it is better practice to write an interface ServiceType, implemented by the Service class, and make the Master class have a dependency on this interface, rather than on the implementing class. The ServiceType interface should provide all the methods from Service that are used by the Master class. This decouples our program and allows us to use different implementations of the ServiceType interface, which are passed into the Master class through its constructor. This approach is the one we?ve taken in the Eggs game from the beginning. All the service classes ? TextMap, ConsoleOutputViewer, ServerHead and so on ? are backed by interfaces, and classes such as StandardGameModel have a dependency on these interfaces, rather than on the implementing classes. The interfaces provide the methods that the StandardGameModel uses to function, but their implementations can have quite different functions: for example, the classes ConsoleOutputViewer and ServerHead both implement the OutputViewer interface; by replacing one with the other, we converted the standalone program into a game server that client programs could connect to. All this came at a small cost, however. In the main methods for our App or EggsServer classes, we needed to instantiate every service class used in our program, passing objects in to other objects as dependencies. Our program is simple enough, but for a more complicated program, keeping track of all the dependencies could turn out to be a major task. Moreover, if we wish to use our classes to create multiple front-ends that have some dependencies in common and some that are different, we?ll need to create helper methods to ?ll in the common dependencies if we want to avoid repeated code. A way to get round this problem is to use dependency injection. This is a pattern whereby a class?s dependencies are ?lled in automatically by means of a con?guration ?le or Java annotations. With dependency injection, we can specify which Dependency injection would ideally be provided within Java itself, but since it is not, we have to use an external framework versions of each dependency we wish to use, and then have these dependencies ?injected? into the classes that need them at runtime. For example, instead of having to pass an OutputViewer object into the StandardGameModel class directly, we can register that we wish to use the ConsoleOutputViewer, and let dependency injection handle inserting this particular dependency into the class when it is required. This means that we do not need to keep track of the inner workings of the StandardGameModel class within our main class. There are two main dependency injection frameworks: Spring and Google Guice. As the README for the Google Guice project says, dependency injection would ideally be provided within Java itself, but since it is not, we have to make do with an external framework instead. In this project, we will be exploring the Spring Framework, but you might like to look at Google Guice as well. Dependency A class or interface used by another class IoC Container The part of the framework that is responsible for running the program and injecting dependencies Bean (@Bean) An object that the IoC container can use to ful?l a dependency Service (@Service) A class that can be instantiated to form a bean Bean Factory Object provided by Spring that creates beans @Autowired This method will be called when a bean is instantiated. The parameters of an autowired method are Spring dependencies @Scope Used to decide whether a bean will be instantiated only once ("singleton") or every time it is needed ("prototype") Spring concepts The core concept in Spring is that of a spring bean. Put simply, a bean is an object that is created and controlled by the Spring Framework. Rather than construct the object using new, we can ask Spring to provide us with an instance of that particular bean. If the bean?s constructor takes parameters, Spring will search for additional beans that it can use to ?ll them in. For example, in our program, we can tell Spring to create a bean of type GameLogic and it will create an instance of the StandardGameLogic classes, as well as instances of all the classes that need to be passed into that class?s constructor. There are two ways to register beans with the Spring Framework. The older way is to write an XML ?le containing data which Spring will read in order to register the bean classes. The newer way, which we shall be using in this issue, is by using Java annotations. If we precede a class declaration with the annotation @Component or @Service, Spring will create an instance of that class whenever it needs a bean of that particular type. For instance, if one of our beans has a dependency on the OutputViewer interface, and we have marked the ServerHead class with the @Service annotation, then Spring will instantiate the ServerHead class in order to satisfy the dependencies of the ?rst bean. The second annotation we can use to create beans is the @Bean annotation, which we place immediately before a public method de?nition. For example, if we write a method returning a Player object, and add the @ Bean annotation to that method, then Spring will call that method and use the return value whenever it needs to satisfy a bean dependency of type Player. If the method takes in parameters, then Spring will look for more beans that it can insert in order to call that method. In order to construct beans, we use a device called a Bean Factory, which is provided by Spring. There are several things that can go wrong in this process. For example, if Spring tries to create a bean of Above Spring cheat-sheet. Spring adds a number of key concepts to the basic Java paradigm in order to model dependency injection www.linuxuser.co.uk 55 Tutorial Java Figure 1 Above Eclipse provides a useful graphical interface for editing the Maven con?guration ?le pom.xml Right In order to import a Maven dependency, we give its name and group ID, and Maven will download the JAR ?les from its repositories type GameModel and ?nds that none has been registered in the project, then it will terminate the program with an error message: is a Maven project in Eclipse, but we have not used the Maven functionality yet so far. Note: in this section the word ?dependency? will refer to an external Java software package which we can add to our project. Everywhere else, we will use it in the sense of dependency injection, as above. If we expand the directory of the ?eggs? project in Eclipse, we see a ?le called pom.xml at the bottom. Double-clicking on this ?le brings up the dialog shown in Figure 1. The pom.xml ?le is responsible for con?guring the Maven build. Eclipse provides this useful graphical dialog which we can use to edit pom.xml. The ?rst change we want to make is to make our project a child of the Spring Framework project. This will make it easier for us to add dependencies without having to specify their version numbers. On the Overview tab, expand the section titled Parent and type Figure 2 A component required a bean of type 'luad.eggs.GameModel' that could not be found. The other problem occurs when we register two beans of the same type in the same program. Then Spring will give us an error message that looks a bit like this: expected single matching bean but found 2 In this case, we have to use further annotations in order to tell Spring which beans to use where. We will not cover this in this issue, but you might like to look up the @Primary and ?λ????? annotations. Installing Spring The ?rst step we need to take is to install the Spring Framework dependencies into our Eclipse project, which we will do using Maven. The project we have provided Java annotations If you?ve never used any kind of Java framework before, you might not have met many, if any, Java annotations. Annotations are used to give the compiler extra info about a particular class, object, method or ?eld. For example, if we mark a method with @Override then the compiler will refuse to compile the module if the method does not in fact override some supertype method. Annotations are more powerful than this simple example suggests. Using the Re?ection API, a software framework such as Spring can query whether or not a method or class has a particular annotation when it is deciding how to handle it. Annotations may also take ?elements?: string data ?elds which can further inform the framework?s behaviour. Just like interfaces and classes, annotations are declared in their own ?les and we have to import them if we wish to use them in another package. Importing annotations in Eclipse is as easy as importing classes or interfaces. If we type the annotation @Service above one of our classes, Eclipse will put an error mark by the side of our code (see Figure 3). If we then click on this error mark, Eclipse will give us the option to import the annotation from the package org.springframework.stereotype (assuming that we have our Spring dependencies installed already). If we double-click on this option, then Eclipse will automatically insert the appropriate import statement into our code. 56 org.springframework.boot into the Group Id ?eld, spring-boot-starter-parent into the Artifact Id ?eld and 1.5.8.RELEASE into the Version ?eld. For the next step, click the Dependencies tab at the bottom, which will bring up the dialog in Figure 2. We need to add two dependencies: Spring boot starter and Spring context. To do these we click the ?Add?? button. For Spring boot starter, type org.springframework.boot in Group Id, and spring-boot-starter in Artifact Id and hit ?OK?. For the second dependency, Spring context, the Group Id is org.springframework and the Artifact Id is spring-context. We don't need to include a Version ?eld, as this is taken care of by the Parent artifact. Last, we need to run the Maven build to install these dependencies. In Eclipse, select Run > ?Run con?gurations?? to bring up the dialog shown in Figure 4. Click on Maven Build at the left, then press the New icon at the top. Under Name at the top, give the new con?guration a name; for example, ?eggs install?. Underneath, click the button ?Workspace?? under the ?Base directory? text ?eld, then click on the name of the ?eggs? project and press Enter. Under Goals, further down, type install. Then, click ?Run? at the bottom to start installation of dependencies. The Maven build will now download the required JAR ?les for the Spring Framework from the internet, if they are not on your computer already. You can check that they have been downloaded by expanding the Maven Dependencies folder for your package in the Package Explorer pane of Eclipse. If it is full of JAR ?les related to Spring, as in Figure 5, then we are ready to get started. Figure 4 Getting started with dependency injection Create a new package in the project called luad.eggs.network.server.springboot, and create two classes in it: EggsServer, which should contain a main method, and EggsServerRunner, which should implement the CommandLineRunner interface from org.springframework.boot. The main method in the EggsServer class is one line long: public static void main(String[] args) { SpringApplication.run(EggsServer.class, args); } We also need to add the following annotation immediately above the line public class EggsServer at the start of the class de?nition: @SpringBootApplication(scanBasePackages = { "luad.eggs" }) Above In order to run a Maven project, we must specify ?goals? for Maven to aim for If you haven?t already, create the method public void run(String... args) inside the EggsServerRunner, and put a simple print statement inside it (e.g., System.out.println("Hello, world!");, so that we can verify that the framework is working. Go back to the ?Run con?gurations? dialog and create a new Maven build, called ?eggs run?. As before, select the ?eggs? project under ?Base directory?, but now type the following under Goals: spring-boot:run When you run the con?guration, you should see output as in Figure 6. Spring produces a lot of output of its own, but among it, the program has printed out Hello, world!. Setting up our beans From a conventional Java programming point of view, this looks odd: our main method hasn?t mentioned any of our other classes at all. What the SpringApplication.run method does is to search for a bean implementing the ApplicationRunner interface or the CommandLineRunner interface and to use the run method of that bean to run the program. So our code to start the application will go in the run method of the =EggsServerRunner class. The annotation @SpringBootApplication tells Spring that the EggsServer class is the entry-point to our Spring application. Then the scanBasePackages element tells Spring that it should search for beans within the luad.eggs package and all subpackages. Since we want Spring to recognise our EggsRunner class as a bean, we must also add the @Component annotation immediately above the declaration public class EggsServerRunner. Figure 3 We now want to mimic what the original EggsServer class does in the Spring Framework, and in order to do that we need to make some more beans for Spring to use. If we look at the loop in the original class, we see that the ?rst main task it has is to create a GameModel object. It does this by creating an instance of the StandardGameModel class, passing in Player, Map and OutputViewer objects that it has created before. Since we are using dependency injection, we don?t want the EggsServer or EggsServerRunner classes to have to de?ne the particular implementations of the GameModel, Player, Map and OutputViewer objects that we are going to use. Instead, we want to be able to tell the EggsServerRunner class to create a GameModel object for each client connection, and then let Spring handle the details of how to set those objects up. For Spring to do its work, we ?rst need to let it know which classes it is allowed to use as beans. In order to do this, we go into the StandardGameModel, HumanPlayer, ServerHead and StreamInputController classes and add the following annotations immediately above the class declaration (i.e., the line public class [...]): XML for dependency injection You might like to try using the older method of dependency injection with Spring, which uses a con?guration ?le written in XML instead of annotations to specify the locations of the beans and their dependencies. There are some advantages to this method, which we have not covered in the main article. For example, it allows you to keep your beans and dependencies entirely separate from your code, meaning that you could effectively change the entire behaviour of your program by swapping out one XML ?le for another. See the Spring documentation at https://spring.io/ docs for more information. @Service @Scope("prototype") The ?rst line tells Spring that it can use this class as a bean to satisfy dependencies. For example, if Spring is trying to create an object of type OutputViewer, Left Importing new annotations in Eclipse is just as easy as importing new classes or interfaces www.linuxuser.co.uk 57 Tutorial More magic with Spring Spring is far more than a dependency injection framework. One useful tool is the @Async annotation, which speci?es that a method should be performed asynchronously; i.e., in its own thread. One advantage of this over new Thread(runnable). start(); is that the method can take parameters. In order to use @Async, we must annotate our main class with the @EnableAsync method. We must also provide a bean of type TaskExecutor ? this is an interface provided by Spring that represents an object that holds a pool of threads and can use them to run different methods asynchonously. Below We can check that our Maven dependencies have installed by looking inside the Package Explorer in Eclipse Java it will create a ServerHead object in order to satisfy that dependency. The second line, @Scope("prototype"), is equally important. By default, Spring beans are singletons: that is, Spring will create the beans it needs when it boots up and will then use the same instances of those beans to satisfy dependencies whenever it needs them. This works well when the beans take the role of services that the entire program can use, such as in the single-player game. However, in the multiplayer game, it is important that we create separate HumanPlayer and ServerHead instances each time we need one. For that reason, we tell Spring to use prototype rather than singleton scoping: now Spring will create a new bean of a given type each time it?s needed and the bean will live for the duration of the life cycle of the object that depends on it. Dependency injection enables us to fully abstract out the networking part of the code to another class Above Spring prints out a lot of its own output to the console, but we can still ?nd our own text output if we look. the TextMap class and the setPosition() method from the HumanPlayer class. Both these methods take parameters: the ?lename of the map and the point to start on. So we open the EggsServer class and add new bean methods to return this information. For example, you might create a method that looks like this: public String getMapFileName() { return "src/main/resources/maps/bigMap.txt"; } The game map, on the other hand, should be a singleton: we want to create a single instance of the map and have all instances of the game model use it. So we add the annotation @Service to the Map class, but we do not add @Scope("prototype") after it. ?and a similar map getStartingPosition() returning an appropriate Point2D object. To tell Spring it can use these methods to create beans, we add the @Bean annotation immediately before these method declarations. @Autowired and @Bean methods Setting up networking There are a couple of steps that need to take place after we create the basic game objects and before we run the game. When we create the game map, we need to call the readMap() method to read in the right map ?le. And when we create a new player, we need to set its position. So far, when we have marked a class as @Service, Spring has only called the constructor of the class when it instantiates it as a bean. Now, we want it to call additional methods too. To do this, we mark the methods we want Spring to call with the @Autowired annotation. In our case, we add the annotation @Autowired immediately before the declarations of the readMap() method from Thanks to the magic of dependency injection, we can fully abstract out the networking part of the code to another class. Create a new class called ClientConnection in the luad.eggs.networking.server.springboot package. This goal of this class will be to provide a bean for the ServerHead class to use in its constructor every time a client makes a connection. Create a ?eld in the class of type ServerSocket called serverSocket and write a constructor that sets this ?eld by taking in a port number as a parameter and passing it to the ServerSocket constructor. Then create a method in the EggsServer class, marked with the @Bean annotation, that returns the port number 9009. The ClientConnection class will use this bean to populate the serverSocket ?eld when it is being set up. Create a second ?eld called clientSocket, of type Socket; create a method fetchNewClientConnection() that sets this ?eld using the serverSocket.accept() method. The fetchNewClientConnection method should return a Boolean value: true if the connection was made successfully (i.e., clientSocket != null) and false otherwise. Now we are in a position to create a bean that will return a Socket object. Create a method in the ClientConnection class called getClientSocket() that returns the value of the clientSocket ?eld. Mark this ?eld with the @Bean and @Scope("prototype") methods so that it will try to return a new value each time. Figure 5 58 Figure 6 Under the hood Dependency injection in Spring works using re?ection: a set of tools that allows us to inspect low-level features of the program, such as the classes and methods it contains. Using re?ection is usually considered a bad idea, since it can make programs hard to debug. However, it also allows us to do things that we would be unable to do otherwise, such as dependency injection. The Java Re?ection API allows us to ?nd all the classes in a package and its subpackages and iterate over all the methods in these classes. (This is what the scanBasePackages annotation element is used for.) These methods are modelled as objects of the special Method class that is part of the Re?ection API. Spring can then call a method getAnnotation() to inspect the annotations of each class and method. For example, if Spring is looking for bean methods, it can call getAnnotation("Bean") on each method and check to see if the value returned is null. If it is not, then Spring can call the getParameterTypes() and getReturnType() from the Re?ection API to ?nd out what type of bean the method provides and what it needs to be invoked. Re?ection can be computationally expensive, so Spring tries to do as little of it as possible. This means creating a cache at the start of the program that holds all the relevant information about which classes, ?elds and methods have which annotations. The ServerHead bean will use this Socket bean to set up its connection when Spring instantiates it. To use the same bean to set up the StreamInputController bean, we will need to create a new bean inside EggsServer that takes in a socket as a parameter and returns its input stream. Mark this bean with @Scope("prototype") as well. Once we have written this bean method, it will use the socket bean from the ClientConnection class to set up the input stream for the server. Since the ClientConnection class itself has singleton scope, Spring will not create a new instance of it each time, instead using the same instance. This means that we will only get a new client connection when we call the fetchNewClientConnection() method. To avoid an error, it?s important that we call this method before trying to instantiate any bean that has a Socket dependency. automatically populate this ?eld with a BeanFactory bean provided by Spring itself. We can now create, for example, a GameModel bean using the code: GameModel gameModel = beanFactory. getBean(GameModel.class); In order to complete the program, remove the print statement from the run() method of EggsServerRunner and replace it with the code to make the game work. First, we need to use our Bean Factory to instantiate a bean of type ClientConnection, which we will use for the networking part of the code. After this, write a while loop that repeatedly calls the fetchNewClientConnection() method from this object and then instantiates two beans of type GameModel and StreamOutputController using the Bean Factory. Last, add the GameModel bean as an Observer of the StreamOutputController object, and call new Thread(controller).start(); to start listening to input from the client. If we look back at the original EggsServer class, we see that the main loop contains two lines of code that we have not yet accounted for: map.addPlayer(player); ?and: player.addObserver(gameModel); Figure 7 Left When we have the server running on the Spring Framework, clients should be able to connect and play the game as they could before Create the game loop Now we are in a position to create the main game loop. We now want to tell Spring to attempt to create beans of type GameModel and StreamInputController each time a client connects. So far, we have seen how to tell beans to create other beans in order to set themselves up, but we have not yet seen how to start up a bean in the ?rst place. In order to do this, we need to use a Bean Factory, which is a special object provided by Spring for the purposes of instantiating beans. Creating a Bean Factory is quite easy. In fact, all we have to do is add the following ?eld declaration to the EggsServerRunner class: @Autowired private BeanFactory beanFactory; Since BeanFactory is part of the Spring Framework, the @Autowired annotation means that Spring will It?s going to be a bit tricky to include these lines of code in our new, Spring-based version of the server. Instead, it makes sense to move them into the constructor of the StandardGameModel class. So we add the following two lines at the end of this constructor: map.addPlayer(player); player.addObserver(this); If you?ve done everything correctly, then the program should run exactly as it did before (see Figure 7). Run the server using the eggs run build con?guration we created earlier and then ?re up a couple of clients in the usual way. If it doesn?t work, use the debugger and the Spring error output to try to ?nd out where you went wrong. www.linuxuser.co.uk 59 Feature Future of Programming Languages THE FUTURE OF PROGRAMMING LANGUAGES Mike Bedford takes a tour of unusual and up-and-coming programming languages to investigate what the future may hold 60 AT A GLANCE Where to ?nd what you?re looking for �Unusual languages p62 Many languages have much in common with so many others but a few have dared to be different. We investigate a handful of languages which break the mould and, in so doing, offer a very different way of programming. Each has something to offer the more adventurous programmer. recent survey of programming languages suggests that programmers are all too predictable. Of those languages used by Linux coders, Python, C, C++ and Java have shared the top few slots for desktop applications for several years and, if we broaden our view to take in web applications, we ?nd that JavaScript and PHP are also popular. With the odd exception, such as the importance of C# on the Windows platform, the list is broadly similar for other operating systems. In one sense, this isn?t too surprising. Considering career prospects, for example, it pays to learn a language that will maximise your employment opportunities A � Languages to watch, p64 �Future languages, p66 envelope, we?d still be using COBOL and Fortran. What?s more, not everyone thinks in the same way, so it?s quite possible that a less popular language could appeal to you and make you more productive. So, if you?re prepared to consider the alternatives, we?re here to help by introducing you to some languages you may not have considered as well as thinking about what the future might hold. A grand tour of programming languages would be a serious undertaking ? one list we discovered contained no fewer than 710 entries ? so we?re restricting ourselves to certain categories and, even then, being very selective. If we?ve managed to whet your appetite, therefore, we trust that you?ll continue this journey of discovery yourself. First of all we highlight a few languages, which, despite not being anywhere close to the top of the league tables, offer a very different way of working and have enthusiastic followings. These languages often tend to favour particular types of application in which they have undoubted strengths and so might not be for everyone, but if your needs match the features offered by one of them, it could be exactly what you need. Even if you don?t have a genuine requirement as a programmer you probably have a general interest in the wider programming landscape so we trust you?ll ?nd this overview absorbing. We?ve referred to these as unusual languages, as indeed they are, although in most cases, that phrase isn?t used in a disparaging sense. Next, we identify the languages that some experts think may eventually topple Python, Java and C++ from the top spots ? these we're calling the up-and-coming languages to watch. Of course, there?s no guarantee that any of these will ever share the success of C++ or Java, and there?s really no sure?re way of predicting which will be the next must-have language. It often takes a long time for a language to receive recognition and gain critical mass. Indeed, the past shows that a long history of lacklustre performance is no indication of a future of mediocracy. Python, for example, was ?rst conceived in the late 1980s and launched in 1991, but it was 2006 before it gained a top-ten position in the TIOBE (The Importance of Being Earnest) Index of the popularity of programming languages. It took a full 26 years to reach its highest-ever position of number four that it enjoys today. Finally, we?ll investigate current research into programming languages, which, despite over 60 years of development for high-level languages alone, is still going strong. Researchers are developing better ways of instructing computers on what we need them to do and we ask a couple of experts to describe the current areas of interest and predict how soon these initiatives will yield a new way of working. The programming scene is in a state of constant change so it's extremely likely that we will be using some different languages in ten years? time. It?s also likely these languages will be starting to gain momentum already, so we take a look at today?s up-and-coming languages. Despite the obvious draw of the most popular languages, there are good reasons to look elsewhere and this tends to mean those that are most commonly used in business. There are also other good reasons to pick the most popular ones for personal use. Perhaps most importantly, the more prevalent a language is, the more likely you are to ?nd adequate community support. Despite the obvious draw of the most popular languages, there are also good reasons to look elsewhere. After all, if nobody had ever dared to push the To get a feel for what the future of programming might hold, we catalogue the development of languages over the past sixty odd years and build on these foundations by presenting the views of those who are involved in leading-edge research into programming languages. w . nuxuser.co.uk 61 Feature Future of Programming Languages Unusual languages Languages that couldn?t be more different from those commonly used today ere we look at languages that can be considered unusual in the sense that they are not commonly used, but many are by no means new. But this doesn?t necessarily mean that they?re failed. Unusual languages are often designed for niche applications or, perhaps, they appeal only to a minority of programmers but still attract enthusiastic support. It?s entirely possible, therefore, that one of the languages discussed here could provide a solution for you or, at the very least, provide an interesting lesson into programming more generally. H Above When ?rst introduced, APL needed a special keyboard and golfball printer to cope with its strange character set. Recent changes amended this, thanks to improved font technology and on-screen keyboards Prolog Prolog might date back to 1972 but, unlike most of the languages used today, which are third generation, Prolog has been described as a ?fth-generation language in some quarters. This is because it is used to de?ne the problem but doesn?t tell the computer how to solve it in the sense of providing an algorithm. Working in Prolog involves de?ning relationships and rules and language concentrate on AI. Despite the fact that this might suggest that Prolog is a language solely for niche applications, it is Turing Complete, which means that it can be used to solve any computable problem, just like Python or C++ ? although realistically you probably wouldn?t want to use it for mainstream applications. A trivially simple example would be to provide the fact de?ned by wine(chardonnay), which means that Chardonnay is a wine, then the rule drink(X) :wine(X), which means that anything that?s a wine is a drink. Now, if you were to issue the query ?- drink (chardonnay), meaning 'is Chardonnay a drink?, Prolog would respond with the answer 'true'. It can be used to solve any computable problem, just like Python or C++ then making queries about the database of these relationships and rules. It soon became the language of choice for arti?cial and natural language applications ? indeed, most introductions to the Given that new programmers learning Java or Python write code to solve trivially simple problems, we trust that you?ll recognise that much more interesting tasks can be solved in Prolog and very sophisticated expert systems can be put together this way. There's no shortage of Prolog implementations, many for Linux, but if you just want to get a feel for the language, a good option is to use the online version at https://swish.swi-prolog.org. COBOL COBOL is included here to illustrate how different some languages are ? even ones intended for mainstream programming. Dating back to 1959, it was once the language of choice for business as opposed to scienti?c programming and it is still the 25th most popular language, but mostly for maintaining legacy systems. So it?s unusual in the sense that it?s almost never used for TIMELINE 1842 1948 1949 1952 Ada Lovelace writes the ?rst computer program, designed to run on Charles Babbage?s un?nished Analytical Engine The ?Small-Scale Experimental Machine? becomes the world?s ?rst stored program computer, programmed in machine code EDSAC is programmed in one of the ?rst assembly languages, Initial Orders. The assembler was hard-wired into it Autocode is the ?rst high-level language. It?s the ancestor of all today?s languages, even though it never took the world by storm 62 QUICK GUIDE Esoteric languages This type of language is never used to solve real world problem but you might ?nd them intriguing, nevertheless. Some were designed as a challenge to programmers because they are so dif?cult to use. Java2K, for example, uses base 11 arithmetic, with space to represent the digit 10, so you can?t make a program easier to read by adding spaces. Furthermore, it?s a probabilistic language rather than a deterministic one (don?t ask). Others, like INTERCAL, are more of a joke. It has no GOTO type instruction but it does have COME FROM, which takes some getting used to. Slightly less perversely, but not a lot, are those esoteric languages designed to be minimalistic so require very simple compilers. Brainf*** (yes, it's a swear word) is a universal language but it has just eight instructions, each represented by a single character. One compiler is just 100 bytes long. Unusual languages are often designed for niche applications or, perhaps, they appeal only to a minority of programmers rate GIVING Gross-price. GnuCOBOL and OpenCobolIDE are a couple of open source COBOL projects, should you want to have a go at this unusual language. APL APL stands for A Programming Language and was ?rst released by IBM in 1964. It can be used to write a program in the sense of a sequence of instructions which is then executed, but it can also be used as a glori?ed calculator, entering instructions to which the result is immediately displayed. This interactive programming environment is one of the main features that draws people to APL, even today. Also unusual is the fact that a lot of computations that would take several lines of code in most languages require just a single statement in APL, e.g. If you were to type 8!, APL would respond with the answer 40320, which is 8 factorial. Matrix manipulation can also be carried out using single instructions. APL uses single characters for many of its operations and this terse nature is both a blessing and a curse. It?s an advantage in that you don?t need to type a lot to get an answer but it?s notoriously dif?cult to learn, let alone maintain code. Not only that but, because the ASCII character set doesn?t contain nearly enough symbols, a unique character set is used. At one time this required a special keyboard, or at least a good memory of which combination of keys to use to access a given character, but many of today?s implementations overcome this by offering on-screen keyboards. To give you a feel for both the power and the seemingly unfathomable nature of APL, the following statement causes all prime numbers up to the value R to be displayed: If this has inspired you to try it out yourself, and perhaps take a degree of satisfaction in mastering such an arcane language, http://tryapl.org is a good online APL implementation. Alternatively, there are several versions available to install on your PC, some of which run under Linux. If our look at these three unusual languages has whetted your appetite to delve into a few others that will never appear at the top of league tables, give Lisp a try ? and, in particular, Common Lisp ? and J, which was inspired by APL but doesn?t use its weird character set. Above APL is able to provide an interactive programming environment CC BY 2.0 Jan Arkesteijn new applications today, although there are still employment opportunities for those pro?cient in this language of yesteryear. The language is very powerful in the way it handles hierarchical data structures, which is very useful in traditional data processing tasks, but its most unusual aspect is its instructions. Called sentences, they have full stops at the end, but this isn?t the limit to its similarity with the English language. The idea was that accountants and others with no expertise in programming could look at a program and understand its function. So, where most languages might use a statement like Gross_price = Nett_price * Vat_rate, in COBOL this would be MULTIPLY Net-price BY Vat- Above COBOL, which was developed by Grace Hopper, looks odd by today?s standards 1957 1958 1958 1959 Fortran is the ?rst mainstream, high-level language, ideal for scienti?c application ? the main use of early computers LISP uses the paradigm of list processing and gains support for AI, it's different in nature from other early high-level languages ALGOL, the ?rst blockstructure language, is designed and encourages improved programming techniques. COBOL is launched for business applications, setting the scene for different languages for different applications. www.linuxuser.co.uk 63 Feature Future of Programming Languages Languages to watch anguages come and go and, if we were to analyse the most used ones over the years, we?d surely come up with a different list for each decade. While still in the top ten, for example, in the last 10 years PHP use has plummeted from a usage ?gure of over ten per cent to less than two per cent. Conversely, Python has increased from just over on per cent in 2002 to almost ?ve per cent today. This begs the question of what we?ll be using in another decade or so. Languages rarely take off in a big way overnight so, almost certainly, the ones that we?ll be using in ten years? time will already have been released and will already be starting to gain popularity. Accordingly, we?re looking here at what we?ve referred L BACK IN TIME Programming the Turing Machine Before real-world digital computers, Alan Turning came up with a hypothetical machine capable of universal computation. It was never designed as to be practical and it wouldn?t have been fast, but the concept was instrumental in developing the theory of computation. Despite it not being a physical computer, it has been simulated and this provides the opportunity for anyone to get a feel for what programming it would be like by heading over to www.turingmachinesimulator.com for a tutorial and an online simulator. to as the up-andcoming languages of the programming world. Coming up with a list of the top languages to watch, would involve making a very subjective decision. Rather than relying on our own judgement, therefore, we?ve distilled the views of several commentators to come up with the top few. Above The TIOBE index: Tracking the rising stars of the programming world We?ve looked to the developers of two that represent quite enables ?exible and modular program different advanced programming concepts construction. Go compiles quickly to to see what they believe sets their creations machine code yet has the convenience apart from the competition. We also of garbage collection and the power of mention a few others that would be worthy run-time re?ection. It?s a fast, statically of consideration if you?re looking for the typed, compiled language that feels like a next big thing. dynamically typed, interpreted language.? You can try Go for yourself online at https://play.golang.org but there are also Go lots of online educational resources which Perhaps the language that most people are would be well worth investigating if you?re getting excited about, Go is an open-source new to the language. Take a look,at the project developed by a team at Google and online tour at https://tour.golang.org. elsewhere. Unlike many up-and-coming Then, once you?ve exhausted the online languages it?s comparatively new, ?rst resources, you?ll probably want to install appearing as recently as 2012. The team Go on your PC. For that, you should visit say that it?s designed to make programmers https://golang.org/doc/install. more productive ? so nothing new there, then ? but in particular they make the following claim: Haskell ?Go is expressive, concise, clean, and Haskell is another open-source project ef?cient. Its concurrency mechanisms boasting over 20 years of development. make it easy to write programs that get According to its developers, it is ?a the most out of multicore and networked polymorphically statically typed, lazy, machines, while its novel type system purely functional language, quite different TIMELINE 1964 1964 1972 1972 BASIC is an easy-to-use ?rst language, designed for beginners. It is used extensively on 1980s home computers APL offers an interactive programming environment. Using a unique character set, it can express problems concisely Smalltalk is one of the ?rst languages to feature object orientation. It in?uenced many of today?s popular languages C is launched. Building on ALGOL, it becomes very popular, eventually giving rise to derivatives like C++ and C# 64 www.tiobe.com The ones that could topple Java, C, C++ and Python A list of the new languages that are attracting attention would be a long list indeed from most other programming languages.? They further suggest that even if you are not in a position to use Haskell in your programming projects, learning it can actually make you a better programmer in any language. Since the functional aspect is, perhaps, what the community seem to suggest is Haskell?s key feature, it would be worthwhile looking in a bit more detail about what this programming paradigm involves. Again, quoting its developers, this sets it apart from languages such as C and Java and many other imperative languages. ?They are imperative in the sense that they consist of a sequence of commands, which are executed strictly one after the other,? they explained before pointing out how functional languages differ. ?A functional program is a single expression, which is executed by evaluating the expression. Anyone who has used a spreadsheet has experience of functional programming. In a spreadsheet, one speci?es the value of each cell in terms of the values of other cells. The focus is on what is to be computed, not how it should be computed.? To make the point, they present a sort program which runs to six lines in Haskell and the C equivalent which occupies 24 lines, both being suitable spaced for clarity. As with many programming languages, you can try Haskell out in your browser, speci?cally at http://tryhaskell.org and you can access a quick tutorial there. There are QUANTUM PROGRAMMING D-Wave machines D-Wave, the manufacturer of the world?s only commercial quantum computer, offers several ways of programming its hardware. These range from common high-level languages, through languages designed to solve optimisation problems ? the main strength of D-Wave machines, to the low-level Quantum Machine Language, the equivalent of the machine code used to program early computers. Employing a so-called quantum annealing architecture, D-Wave machines can be thought of as analogue quantum computers. They are being used by Google and NASA to solve certain types of problems but they are not universal as our digital computers are. Meanwhile, Above New programming languages are being developed for quantum computers also several compilers available for various operating systems including Linux. Consider also? A list of the other new languages that are attracting attention would be a long list indeed. However, if you want to continue this tour of discovery, the following languages are well respected: Kotlin, Scala, TypeScript, Clojure, Rust, Swift and Julia. Drawing up an ordered shortlist would be tricky but if you want to try your hand at a language that?s both up-and-coming and unusual, in the sense of employing a researchers are intent on bringing us universal, digital quantum computers, with the phenomenal power they will provide, and languages to support this hardware are already being developed. For more than a decade, a team of researchers at Microsoft has been working on universal quantum computing, a model of computation that employs qubits instead of bits. Qubits can store binary 0s and 1s simultaneously, which could potentially lead to hugely parallel processing. Needless to say, using this bizarre sounding architecture requires a very different approach to programming. Just a couple of months ago, Microsoft announced a new language designed for developers to create apps to debug on quantum simulators today and run on an actual topological quantum computer in the future. Fortuitously, they say that you don?t have to be a quantum physicist to use the new technology. According to a company spokesperson, ?The new programming language is deeply integrated into Visual Studio, and it includes the kinds of tools that developers rely on for classical computing, such as debugging and auto complete.? somewhat different programming paradigm, how about trying Clojure? As a dialect of the old Lisp language, introduced back in 1958, it is a so-called list processing language. Quoting Rich Hickey, Clojure?s author, ?Clojure is a dynamic, general-purpose programming language, combining the approachability and interactive development of a scripting language with an ef?cient and robust infrastructure for multithreaded programming. Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system.? 1991 1995 2009 2017 Building on BASIC, Visual Basic is designed for ease of use, tailored for developing programs for GUIs PHP and JavaScript appear. Improving facilities for serverand client-side programming, they bring more interactivity Go is introduced with support for concurrent programming. It?s one of today?s top up-andcoming languages Microsoft releases a language for quantum computers, maybe the most signi?cant progress in programming for 60 years www.linuxuser.co.uk 65 Feature Future of Programming Languages Future of programming What does the future hold for programming languages? efore looking at the future of programming languages, it would be helpful to delve into the past so we can understand the context. The very ?rst computers were programmed in machine code. Initially, the program was entered by writing binary numbers to speci?c addresses using switches on the computer?s front panel, although punched paper tape was soon used instead. For example, to jump to address 0011100 on an old 8-bit processor, you might have to enter the op-code (the binary number that represents the instruction) 00001110, to one particular address and 0011100 into the next address. Needless to say, this wasn?t at all intuitive by today?s standards and was very errorprone. First of all you needed to look up the op-code correctly, and then you?d need to accurately work out the address to jump to. What?s more, maintaining or modifying code wasn?t at all simple. For instance, if you added an instruction somewhere in the code, all subsequent instructions would have to be written to different addresses and any jump or branch instructions that B Java Fortran C with classes ALGOL BCPL B 66 C Above Today?s programming languages, such as C++, are the result of evolutionary developments over 60 years. might change from 00001110 0011100 to JPM INITIALISE. Not only was this much easier and more intuitive to write and read, but code maintenance was less error-prone since the addresses referred to by labels would change automatically as instructions were added or removed and, similarly, the addresses referred to by variable names would also change as variables were added or removed. Even so, an inherent drawback with machine code and assembly language is that the set of instructions available is unique to a particular computer architecture. This means that programs written for one machine couldn?t run on another. A single high-level instruction would often replace several machine code or assembler instructions with clear bene?ts referred to those instructions would be to be modi?ed to refer to their new addresses. Next came assembly languages, which were designed to overcome the drawbacks of machine code. This was achieved using software called an assembler that would translate assembly language code into machine code. This provided several advantages. First, instead of using an opcode, a meaningful mnemonic was used. In addition, variable names were used to refer to memory locations and instruction could be identi?ed by labels. That ?rst generation jump instruction, therefore, C++ QUICK TIP The patient programmer Early programmers were patient. They wrote their programs on a stack of punch cards and gave them to an operator. A couple of hours later, they?d be told there was a 'syntax error'. Above We?ve come a long way from entering machine code via a computer?s front panel switches Furthermore, learning to programme on one computer didn?t stand you in good stead for programming on another. The next major development was of high-level languages, which were designed to overcome the issue of non-portability that applied to machine code and assembly languages. These languages have also been referred to as third-generation languages (and, retrospectively, machine code and assembly languages became ?rst- and second-generation languages), although the generation terminology isn?t recognised by all programming language researchers. Instead of using the built-in instructions of a particular machine, high-level languages provide a general set of instructions that would be translated by a compiler into the instructions supported by the target machine. Other bene?ts were also on offer as the term ?high-level? suggests. In particular, a single high-level instruction would often replace several machine code or assembly instructions with clear bene?ts to the programmer. So, for example, to add the contents of two memory locations with the variable names A and B, storing the result in memory location B, might require the following three assembly instructions: LOAD A, ADD B, STORE B. In a typical high-level language, these three instructions would reduce to a single instruction such as B = A + B. The concept of fourth- and ?fthgeneration languages goes back quite some time but these tend to be somewhat specialised and the vast majority of the languages now in use for general-purpose programming are third generation. Because the phrase isn?t universally accepted and, according to one expert, is mostly used as a marketing gimmick by suppliers of some Q&A Interview: Martin Lester, University of Oxford Above The punch card epitomises the early days of programming in languages like Fortran and COBOL programming by allowing code to be arranged in blocks by use of the BEGIN and END statements. This soon gained widespread support and led to several spin-off languages, most importantly, from our perspective, a now largely forgotten language called BPCL which led to B which, in turn, inspired C. Although undoubtedly a high-level language, C added low-level features such as bit manipulation functions. This made it suitable for ef?cient system programming, but it gained widespread support far beyond this initial niche. C++ built on C by adding object orientation and this, in turn, was a direct predecessor to Java, which has a similar syntax to C++ but introduced the concept of the virtual machine for the ultimate in portability. Testing is a good way of checking out a program but it?s impossible to ?nd every bug languages, we won?t look in detail at how fourth- and ?fth-generation languages are de?ned. However, if we do think in terms of generations, the general consensus is that each generation aims to provide a means of de?ning a problem that is more removed than the previous generation from the workings of the hardware as we?ll see later. Evolutionary development Machine code and assembly language are different because they were tied to particular computer architecture. However, with the introduction of high-level languages we can trace evolutionary paths that link today?s commonly used languages with some of the very ?rst ones, dating back to the late 1950s. As an example, we?ll trace the ancestry of Java, today?s most used language. The lineage is based on a study undertaken by IBM Research, the Retrocomputing Museum, Microsoft and Stanford University. Fortran, the ?rst high-level language to achieve popular support, was designed for scienti?c programming. Flow control was pretty much limited to GOTO, DO loops, functions and sub-routine calls and this led to 'spaghetti code'. ALGOL addressed this issue by providing support for structured The way ahead To get a view for what the future may hold, we spoke to two experts in programming language research ? Professor Colin Runciman of the University of York and Dr Martin Lester of the University of Oxford. A key issue raised by both is those developments aimed at making code more reliable. Testing is a good way of checking out a program but it?s impossible to ?nd every bug this way and errors often only come to light when the software has been distributed. Static analysis ? detecting errors at compile time ? is also limited. Type systems are a classic way of discovering errors at compile time and a lot of current research is being conducted in this area with the aim not just of ?nding obvious errors like adding a text string to an integer, but less obvious ones such as writing a loop that will never terminate. One product of this line of research has been optional static type systems for languages that are traditionally dynamically typed. Examples include TypeScript for JavaScript and mypy for Python. Martin Lester is a researcher at the University of Oxford with an interest in the theory and application of programming languages. Today?s top languages are mostly general purpose, imperative and object-oriented, so does that paradigm represent the pinnacle of development? Just because a language is popular and widely used, it doesn?t necessarily mean that it?s been chosen for a good reason, or indeed chosen at all. If a developer contributes to an existing project, the language is already ?xed. Most professional developers work on existing projects, so get no choice in what language they use. So are you saying there is more to come? C++, Java and Python have all undergone a lot of changes in the last ten years or so. Whenever a new feature is added to a language, the bene?t is usually one of being able to write programs more quickly, have them run more quickly, or reduce the number of errors in them. So although I don?t know what new features or paradigms are going to arise, I?m fairly con?dent that those will be the bene?ts. What progress is being made in expressing problems in natural languages like English? The problem is that our use of language is inherently ambiguous and contextual. If you try to have a discussion with even the best online chatbot, you?ll probably ?nd them pretty lacking ? although they usually manage, or at least try, to cover this up by being deliberately vague and noncommittal. If we ever manage to solve the hard AI problem of interpreting natural language, it won?t matter how good our program synthesis is or which programming language we use, as the same AI will be able to write programs in whatever language we ask them to. However, until that time, we?re de?nitely still going to need human programmers. www.linuxuser.co.uk 67 Feature Future of Programming Languages Q&A Interview: Professor Colin Runciman, University of York Colin Runciman works at the Department of Computer Science at University of York with research interests in programming languages and systems, functional programming and software tools. Above The success of Amazon?s Alexa tech might suggest that engaging with computers using natural language may replace programming. But making queries is one thing; solving computational problems is quite another A second area is support for the increasingly parallel nature of computers. ?On the one hand, you have the scienti?c computing community, who are looking at how best to shift their computation onto massively parallel GPUs using CUDA or OpenCL,? says Lester. ?On the other hand, people try to write their server applications to scale well to multi-core, multi-processor systems, whether in their own data centres or in the cloud. Two languages I hear people talking about a lot here are Go and Rust, which have different approaches to memory-safe concurrency. Go supports this based on having many processes that share data by passing messages, rather than by sharing programming, namely constraint: ?Most of the research is focused on how to ?nd solutions rather than on developing the language in which to describe the solutions. There is a wide range of constraint solvers and a correspondingly large range of input languages. Unfortunately, some of the solvers are really good at solving certain kinds of problems but hopeless at others. So if you want to use constraint programming for a particular problem, you might have to try several solvers to ?nd one that works well. ?The SMT-LIB project has produced a standard input language for constraint solvers, which all the major solvers support, and a library of benchmarks, so you can try to compare them and ?nd out which works best for your problem. But the input language is quite low-level, so what often happens is that people write a program to generate the input. In terms of theoretical developments, researchers will doubtless make their solvers better at solving harder problems more reliably, but I will be more interested to see to what degree constraint solvers become used as part of mainstream applications software.? And, ?nally, if you?re worried that current research might put you out of work as a programmer, fear not ? neither of the experts believe that programmers will be on the scrap heap anytime soon. Neither of the experts we interviewed believe that programmers will be on the scrap heap anytime soon variables and using locks. Rust, meanwhile, has a clever compiler that checks shared variables are used safely.? Needless to say, this is a major area of ongoing research. More fundamentally, though, another strand of research mentioned was declarative programming, where you state the problem as opposed to de?ning the sequence of instructions needed to solve it. Lester explained the challenges of research into one class of declarative 68 Given that many of today?s languages date back many years and their roots go back 50 or 60 years, have high-level languages reached maturity or are there further developments to come? Further developments should be expected in at least two areas. First, stronger type systems will increase static guarantees that certain kinds of failure cannot occur when the program runs, and that many desired correctness properties are veri?ed. Second, more powerful methods will be developed to compile programs effectively for execution on multi-and many-core machines, including strategies for portable optimisation. The terms fourth and ?fth generation date back quite some time but such languages haven?t made a huge impact. Is this going to change? The impact of declarative programming, mainly functional and logic programming, is set to increase in my view. Since some languages don?t involve creating something that could be called an algorithm, are the days numbered for programmers? Will queries eventually be made just by expressing questions in natural language? Query languages are one thing ? languages for solving computational problems are quite another. Many queries can already be cast quite effectively in natural language, even if it is just a Google search. But natural language would be hopeless for programming something like a network communication protocol, a web browser, an optimising code generator, or a task scheduler in an operating system. We need programmers who are highly trained in the fundamentals of software composition, including algorithms, data structures, components and the principles of software correctness and portability. US subscription offer Get 6 issues FREE When you subscribe* U SPINS T N U B U D TRY 2 V D E E R k F uxuser.co.u The open source authority for enthusiasts and developers www.lin ER LINUX US E 186 PER ISSU & DEVELO AZINE NTIAL MAG THE ESSEGNU GENERATION FOR THE ULTIMATE & E U C S RE KIT R I A P E R system very � File Data reco Security analyysis GUIDE IN-DEPTH f re o The futu mminsg progra t language The ho to learn PAGES OF GUITDproEtoS col Io aster the HTTPS > MQTT: M Intercept y: it ur ec S > of Sed y jo e l Linux: Th > Essentia IN THE PRINTED UK £6.49 ISSUE 186 FREE resource downloads in every issue O INSIDE r Pro2 hu age DiskAs mework : Spri g Fra -Fi Wi ef eli r e ste 7 15:10 29/11/201 Offer expires 31 January 2018 Od h Order hotline li +44 344 848 2852 Online at www.myfavouritemagazines.co.uk/sublud *This is a US subscription offer. 6 free issues refers to the USA newsstand price of $16.99 for 13 issues being $220.87,compared with $112.23 for a subscription.You will receive 13 issues in a year.You can write to us or call us to cancel your subscription within 14 days of purchase. Payment is non-refundable after the 14 day cancellation period unless exceptional circumstances apply. Your statutory rights are not affected. Prices correct at point of print and subject to change. Full details of the Direct Debit guarantee are available upon request. UK calls will cost the same as other standard ?xed line numbers (starting 01 or 02) are included as part of any inclusive or free minutes allowances (if offered by your phone tariff). For full terms and conditions please visit: bit.ly/magtandc. Offer ends 31 January 2018. T IME T O S T E P OF F T H AT T R E A DMIL L With so many demands from work, home and family, there never seem to be enough hours in the day for you. Why not press pause once in a while, curl up with your favourite magazine and put a little oasis of ?you? in your day. To find out more about Press Pause, visit; pauseyourday.co.uk THEESSENTIAL GUIDE FOR CODERS & MAKERS PRACTICAL Raspberry Pi 72 ?Using very small robots to teach robotic behaviour? Contents 72 Pi Project: the micro robots are truly tiny 74 Draw vector shapes in your Minecraft world 76 Add voice-activatedAI withGoogleAssistant 78 Create a lightweight database using SQLite www.linuxuser.co.uk 71 Pi Project Micro ro ots Micro robots A pragmatic approach to teaching robotics has led to a project to build robots smaller than pocket change Joshua Elsdon is currently a PhD candidate at Imperial College London, studying in the ?eld of handheld robotics and augmented reality. He?s been a keen tinkerer all his life, starting with audio engineering and high-voltage circuits as a teenager which has developed into a passion for robotics. Like it? Head to Joshua?s YouTube channel for demos of his camera-based location system and very tiny robots following lines, performing synchronised patterns and generally whizzing about: http://bit.ly/ JEmicrorobots Further reading To keep up to date with Joshua Elsdon?s micro robots project and the release of version 6 of the kit for under £100, head to http://bit.ly/ HdayMicroRobots 72 Can you give us an overview of the micro robots projects? What?s the idea behind your micro robots? The micro robots project was formed when discussing how the Imperial Robotics Society could develop a course for teaching higher-level multi-robot behaviour. We have a very successful course introducing the basics of robotics on a small robot platform, roughly the size of an A5 sheet of paper, but robots of this size quickly become a problem if you want to control a load of them at once. The area you have to cordon off becomes prohibitive; also, generally you can only have one set that the class must share. We decided that this course would not need access to the low-level hardware, as that would have been covered in the previous course, so we can use the full power of miniaturisation to reduce cost and size. We hope that in using very small robots to teach robotic behaviour We use the robots for fun programming exercises classes, we can have multiple systems available for testing and have to use less space for the arenas. Additionally, the low cost of highly integrated electronics that could be assembled automatically could lower the burden on volunteer instructors. Naturally, this seed for the project has given rise to a multi-year development effort for me and my hobby partner Dr Thomas Branch. You?ve recently mentioned using a camera, QR and OpenCV for tracking the robots ? can you explain how this works? For most robotic experiments, knowing the location of individual robots is a fundamental piece of information. The issue is that the sensors on board the robots are not sophisticated enough to discover their location from the environment. So a typical solution is to have an overhead camera keep track of where the robots are to provide input to the navigation algorithms. With a ?xed camera this can be achieved reasonably simply as the system?s coordinates can be based relative to the camera and the size the robots will appear in the image can be hard-coded. Though due to the fun I have had whipping the robots out at opportune moments, I wanted the system to be possible to deploy completely ad hoc. Therefore, we have implemented a system that uses a QR code like marker in the scene as a coordinate reference that provides the computer vision system with a sense of scale and orientation. The camera does not need to be orthogonal to the surface, and we can even use the location of the camera as input to the system. You also mention using the Raspberry Pi 3; how does that ?t into project? Originally we were thinking of this as a business case for providing educational kits, which are very price sensitive. Using the Raspberry Pi jumped out as a method of supplying the complete computational system with no setup for the user. We were aiming for the cost price of a robotic system with six robots and master computer to be roughly £100. Though because we are still doing lots of development on the project, we primarily use a full desktop system, for convenience. Have any interesting projects have come out of the micro robots project and the training you?ve been running at Imperial? Currently the robots are not used in teaching at Imperial, though in the future we hope to change that. I am using them in my private tutoring sessions with two 13-year-old boys. We use the robots for fun programming exercises, and we use a larger version of the robots for teaching SMD soldering techniques. The primary guiding project is to implement robotic football, though I always try and let the students have input on where the project goes, so we will have to wait and see what we actually implement. Can you tell us about the robot HAT you?re working on? We had a couple of spare pins on the programming connector for the robot, so we decided to break out an I2C interface for expansion boards. As a proof of concept, we are partially through implementing a TOF (time of ?ight) laser scanner on the front of the robot. Due to the precise nature of the stepper motors we use for the drive motors, we should be able to collect a 360-degree laser scan of the environment from a robot. This can be used for SLAM (simultaneous location and mapping) which, if we can pull it off, would be by far the smallest robots to complete this task. You mention that v6 is ready for manufacture? Is there a kit coming out soon? Yes V6.0 is more or less ready to go; it implements an accelerometer for our new idea, running the robots on walls. We have demonstrated the fact that the robots can drive on a ferromagnetic surface mounted to a wall; the accelerometer will provide all robots with a reliable absolute orientation relative to gravity. As far as kits, it seems unlikely that there would be a kit any time soon ? everything you need to know is open source; only the batteries are a pain to get. We are likely to make a larger batch this year for a demonstration of the system, and perhaps that would lead to some opportunity where the robots can be supplied publicly. Custom wheels Components Robot Q STM32F031 Q 2Χ forward-facing IR proximity sensors and 1Χ downward-facing IR line sensor Q 2Χ micro stepper motors Q IR uplink and downlink modulated at 38kHz System Q ST-Nucleo based IR bridge for communication between master and robots Q Master Linux system (RPi or laptop) Q User input such as joystick ?The project took a leap forward when we committed to the idea that we needed to manufacture our own wheels for the robots,? says Joshua. He was originally using a friction transmission from the motors to the wheels, as the wheels he could buy didn?t have any gear teeth on them. He also built a DLP (Direct Light Processing) projector-based 3D printer, which, he says, enabled him to control everything to make high-quality wheels to their required speci?cation. Master Pi The project uses a master computer to control the micro robots and to reduce the cost of a kit. The team has experimented with various singleboard computers, including the Raspberry Pi Zero W and the Orange Pi Zero 2 Plus, primarily with the aim to make the master system ?more pocketable,? says Joshua. ROS junkies Robot Operating System?s (ROS) visualisation tools, debugging tools, repositories of state-of-the-art algorithms and communication simplicity ?were just too good to turn down,? says Joshua. ?Through my research work, myself and most of my colleagues are ROS junkies, so for me it wouldn?t count as a real project unless it can leverage the ROS ecosystem.? Above Version 6.0 of the micro robot kit is almost ready to go (rendered, above) and includes an accelerator for running the robots on walls: ?We have demonstrated the fact that the robots can drive on a ferromagnetic surface mounted to a wall,? Joshua tells us. ?The accelerometer will provide all robots with a reliable absolute orientation relative to gravity.? Arduino drop The project used the ATmega328p at ?rst, but the team soon got frustrated with the lack of resources in the Arduino IDE and so switched to the STM32L031. This offered plenty of timers to implement better motor control and have more ?ash and RAM. ?[As] these robots are meant to abstract away from the low-level details for the user, using Arduino was probably misguided in the ?rst place,? says Joshua. Above A recent trip to a large robotics conference saw the micro robots well received, but did highlight a few problems: ?The lesson learned [?] was that any demo of the project should be portable and quick to set up,? says Joshua. For future trips he intends to integrate his calibrated system with controlled lighting and a mount for the camera into a single-board computer www.linuxuser.co.uk 73 Tutorial Minecraft Drawing vector shapes in Minecraft with Python code Calvin Robinson Let?s build some shapes in Minecraft using nothing but a few lines of Python code and the handy McPiFoMo hook is Head of Computer Science at an all-through CofE state school in Barnet. Calvin also consults with schools all over London, providing highquality Computing curricula. Just make sure you pop it in the ~/home/.minecraft directory along with McPiFoMo. Another way of installing ?Minecraft Stuff? would be to use Python?s package index tool with sudo pip install minecraftstuff or sudo pip3 install minecraftstuff. 02 We?re going to want to create a new .py ?le in our favourite text editor / IDLE and import all the relevant Minecraft related Python modules. Resources McPiFoMo http://rogerthat. co.uk/McPiFoMo.rar Block IDs http://bit.ly/ MC-BlockIDs Minecraft Stuff http://bit.ly/ MC-stuff This issue we?re looping back to where we started with the Python coding in Minecraft series in LU&D178. We?ll be using Python code to draw shapes directly in our Minecraft worlds. This tutorial is written with Minecraft Pi Edition in mind, but you don?t have to be running Minecraft on a Raspberry Pi to follow along. We?ve put together a package that will work on any version of Minecraft, so you can run this tutorial on your favourite ?avour of desktop Linux, Pi or no Pi. To allow Python to hook into Minecraft, you?ll need to install McPiFoMo (see Resources) by extracting the contents of the .minecraft directory into ~/home/.minecraft. McPiFoMo includes MCPiPy from MCPiPy.com and Raspberry Jam, developed by Alexander Pruss. Provided you have Python installed, no additional software is required, other than your favourite text editor or Python IDLE. Martin O?Hanlon of stuffaboutcode.com has put together some prefabricated shape functions which we?ll be using in this tutorial; they are available on GitHub (see ?Minecraft Stuff? link in Resources). Python scripts in this tutorial should always be saved in ~/home/.minecraft/mcpipy/, regardless of whether you?re running Minecraft Pi Edition or Linux Minecraft. Be sure to run Minecraft with the ?Forge 1.8? pro?le included in McPiFoMo for your scripts to run correctly. 01 Prerequisites Get yourself a copy of ?Minecraft Stuff?: git clone https://github.com/martinohanlon/ minecraft-stuff Or just visit GitHub and manually download minecraftstuff.py, as that?s all we?ll need for this tutorial. 74 Python prep import import import import import 03 mcpi.minecraft as minecraft mcpi.block as block server time minecraftstuff Connecting Python to Minecraft We?ll want to connect to our Minecraft world: mc = minecraft.Minecraft.create(server.address) Here we?re creating an instance with the variable mc that we can use later on to spawn shapes directly into our open world. We?ll want to use this variable when initiating an instance of ?Minecraft Stuff?: mcdrawing = minecraftstuff.MinecraftDrawing(mc). Now let?s track our current location in-game and we?re good to go: playerPos = mc.player.getTilePos(). 04 Spawning shapes The prefabricated shapes we have available to us are: drawLine, drawSphere, drawCircle and drawFace. We can spawn these shapes in our world with the mcdrawing variable we set up a moment ago, i.e. mcdrawing.drawSphere(), but we?ll need to make sure we pass the coordinates to the function, as well as the type of blocks we?d like to use. For example: mcdrawing.drawSphere(playerPos.x, playerPos.y, playerPos.z, 15, block.DIAMOND_BLOCK.id) 05 Functions and their parameters We have four different shape functions: mcdrawing.drawLine(x1,y1,z1,x2,y2,z2,blockID) mcdrawing.drawSphere(x,y,z,radius,blockID) mcdrawing.drawCircle(x,y,z,radius,blockID) mcdrawing.drawFace(shapePoints,True,blockID) We?ll expand on the drawFace function in a moment. It?s a great tool for ?lling in surfaces. Think of drawFace as a paint bucket from Paint / Photoshop. mcdrawing.drawLine(playerPos.x, playerPos.y + 2, playerPos.z, playerPos.x + 2, playerPos.y + 2, playerPos.z, block. DIAMOND_BLOCK.id) Combined with drawFace, this provides some powerful tools to get creative in our Minecraft worlds, without spending too much time placing blocks. 09 A step in time 10 Status update We imported the time module back in Step 2. That?s so we can slow things down (or speed them up) when needed. When using the drawLine or drawFace functions, it?s best to include a slight pause after each line, otherwise your game could have problems playing catch-up and the lag would result in a messy creation. Just insert a time.sleep(x) after each instance, where x is an integer. A number between 1 and 5 should suf?ce. 06 Custom vectors with drawFace ? part 1 The prefab shapes are great, but you might want to draw a custom shape of your own. We can do that by setting coordinates of each point and then ?lling in the blanks with drawFace. shapePoints = [] shapePoints.append(minecraft.Vec3(x1,y1,z1)) shapePoints.append(minecraft.Vec3(x2,y2,z2)) shapePoints.append(minecraft.Vec3(x3,y3,z3)) 07 Custom vectors with drawFace ? part 2 Now that we?ve set the coordinates of the points, drawFace will connect up the dots and ?ll within the lines with your chosen blockID. mcdrawing.drawFace(shapePoints,True,blockID) For example: mcdrawing.drawFace(shapePoints, True, block.DIAMOND_BLOCK.id). However, if you merely want to draw the outline of the shape, you can change the Boolean from True to False and drawFace will create lines but not ?ll them in. 08 Crossing the line Sometimes you might want to literally just create lines in any given direction, with a speci?c type of block. We can do that with the drawLine function: As your Python script becomes more complex with the addition of different prefab and custom shapes, it?s good practice to comment the code, but also a nice touch to update the current player with a status update. mc.postToChat("Spawning X_Shape in the player world") ?where X_Shape is the name of what you?re spawning. Insert these lines before (or after) creating each shape. Python for Minecraft Pi Using Python, we can hook directly into Minecraft Pi to perform complex calculations, alter the location of our player character and spawn blocks into the game world to create all kinds of creations, both 2D and 3D. We can program pretty much anything from pixel-art, to chat scripts that communicate directly with the player. In this issue we create shapes, both prefabricated and completely custom vector graphics, that spawn in our world at the drop of a hat. With each issue of LU&D we take a deeper look into coding Python for Minecraft Pi, with the aims of both improving our Python programming skills and gaining a better understanding of what goes on underneath the hood of everyone?s favourite voxel-based videogame. www.linuxuser.co.uk 75 Tutorial Google Assistant and ReSpeaker pHAT Raspberry Pi AI Use Google Assistant with your Raspberry Pi to endow it with voice-activated artificial intelligence Nate Drake 02 Return to the Terminal on your Pi and run git clone --depth=1 https://github.com/respeaker/ seeed-voicecard. Switch to this directory with the command cd seeed-voicecard, then launch the installer with sudo ./install.sh 2mic. Once installation is complete, reboot the Pi by running sudo reboot. Reopen a Terminal and run the command aplay -l to is a technology journalist specialising in cybersecurity and Doomsday Devices. list all hardware playback devices. You should see the ReSpeaker listed under card 1 as ?seeed2micvoicec?. Next, run the command arecord -l to list all sound capture devices to check that ?seeed2micvoicec? is listed here too. Resources ReSpeaker Pi drivers http://bit.ly/ GHReSpeaker ReSpeaker 2-Mics pHAT http://bit.ly/ ReSpeakerpHAT ReSpeaker documentation http://bit.ly/ ReSpeakerWiki Tutorial ?les available: ?lesilo.co.uk Google Assistant is a virtual personal assistant, designed to make your life easier by scheduling meetings, running searches and even displaying data. In April this year Google released an SDK (software development kit), allowing developers to build their own Assistant-related hardware. In this guide you?ll learn how to integrate Google Assistant with your Pi and run a demo which can respond to your voice commands. As the Pi doesn?t have a built in microphone, we?ve used the ReSpeaker 2-Mics Pi HAT, which is designed speci?cally for AI and voice applications. The ReSpeaker HAT costs just £12 and is compatible with the Pi Zero as well as the Pi 2B and 3B. However, if you have a USB microphone, you can use this instead. By default, Google Assistant only responds to commands when prompted with a hotword ? ?Hey Google!?. The privacy conscious will be pleased to hear that you can also program the ReSpeaker HAT to listen only when the on-board button is pressed. See the online documentation for tips on how to do this. This tutorial assumes that you have a clean install of the most recent version of Raspbian (Stretch) on your Pi. Once the install is complete, make sure to open Terminal and run sudo apt-get update then sudo apt-get upgrade to bring your system fully up to date. 01 03 Con?gure sound device 04 Adjust sound levels Right-click the volume icon at the top right of your screen and make sure that the voicecard is selected. Right-click once again and choose ?USB Device Settings?. In the new window which opens, click the drop-down menu marked ?Sound Card? and choose the seeed voicecard or another preferred audio device. Next, click ?Make Default? to ensure the Pi will keep using this device next time it restarts. Don?t worry for now about the lack of volume controls; simply click ?OK? to save and exit. Return to the Terminal and run the command sudo alsamixer. This handy utility lets you adjust the volume for playback and recording of all hardware devices. First press F6 and use the arrow to select your chosen audio device, which should be listed under Card 1. Use Return to select it, then press F5 to list all volume settings. Use the left/right arrow keys to select different controls and up/down to adjust individually. Press Esc to quit, then run sudo alsactl store to save the settings. Connect the ReSpeaker Mount the ReSpeaker 2-Mics Pi HAT on your Raspberry Pi, making sure the GPIO pins are properly aligned. If connected properly, the red LED will illuminate. Connect your headphones or speakers to the 3.5mm audio jack on the device. If your speakers aren?t powered, connect another micro-USB cable to the power port on the ReSpeaker to supply enough current. The microphones themselves are located at either end of the HAT, labelled ?Left? and ?Right?. 76 Install device drivers 05 Visit Google Cloud Platform In order to interface your device with Google Assistant, you?ll need to have a Google account. If you don?t have one, head over to https://gmail.com in your web browser of choice and click ?Create an Account? at the top right. Once you?ve done so, visit https://console.cloud.google.com/project and sign in with the account details. Click ?Yes?, then ?Accept?. Next, visit http://bit.ly/GoogleCP. Click ?Enable?. A page will appear entitled ?To view this page, select a project.? Click ?Create? and name your project. Make a note of the project ID, then click ?Create? again. 06 Create product In order to use the Google Assistant API, you need to generate credentials. These are stored in your Pi and let Google know you?re using an authorised device. Click ?Enable? on the API page which has now loaded. Next, visit http://bit.ly/GCP-OAuth. Click ?Con?gure Consent? at the top right and enter a product name such as ?Jane?s Pi Voice Assistant?. If you?re creating a marketable application, ?ll in the homepage, logo and privacy policy ?elds. Click ?Save? at the bottom when you?re done. 07 www.googleapis.com/auth/assistant-sdk-prototype --save -headless. Make sure to replace client_secret_.json with the actual name of the JSON ?le. 09 Enter con?rmation code The Google OAuth tool will now ask you to visit a URL to obtain a con?rmation code. Right-click and copy the link. Open your web browser and right-click the address bar and choose ?Paste and Go?. Click on your own account name and then choose to ?Allow? the voice assistant. The page will generate a code. Copy this from your browser and paste it into the Terminal prompt. If successful, you?ll see a message stating that your credentials have been saved. 10 Start Google Assistant demo In Terminal, run the command sudo apt-get install pulseaudio, then launch it with pulseaudio &. Finally, run the command google-assistant-demo. Make Create client ID Google will return you to the ?Create Client ID? screen. Under ?Application Type?, choose ?Other?, then click ?Create?. You will now see your Client ID and secret key. Ignore these for now and click ?OK?. Google will now return you to the ?Credentials? screen where your client ID will now be listed. Click the down arrow on the right-hand side to save the credentials ?le with the extension .json. Once the ?le is downloaded, open your ?le browser and move it to /home/pi. This will make it easier to access. sure that your headphones or speakers are connected. Use the default hotwords ?Hey Google? to have the Assistant listen, then ask it a standard question such as ?What?s the weather like in London today?? to hear a mini weather forecast. Google Assistant can also play sounds; e.g. ?What does a dog sound like?? It also performs conversions such as asking telling you the value of 100 euros in dollars. Using virtual environments 08 Authorise Google Assistant In your web browser, visit this address: https://myaccount.google.com/activitycontrols. Make sure to switch on ?Web & App Activity?, ?Device Information? and ?Voice & Audio Activity?. Next, open Terminal and run sudo pip install --upgrade google-auth-oauthlib[tool]. You can use this tool to authorise Google Assistant to work with your Pi, using the JSON ?le you downloaded in the previous step. To do this, run the command: google-oauthlib-tool --client-secrets /home/pi/client_secret_.json --scope https:// If you want to use Google Assistant in an embedded device or application, you may wish to create a dedicated virtual environment to store the above demo separately to other projects. Use Terminal to run sudo apt-get install python3.4-venv. Next, choose a name for your virtual environment and create it e.g. python3 -m voice env. If you decide to use a virtual environment, run the above commands after completing Step 7. You can switch to the virtual environment any time with the activate command e.g. source voice/bin/activate. Run deactivate to exit. To run a speci?c program you?ve installed in your virtual environment, enter the command as normal; e.g. google-assistant-demo. www.linuxuser.co.uk 77 Column Pythonista?s Razor Managing data in Python When the amount of data you need to work with goes beyond easy ?at ?les, it's time to move into using a database and a good place to start is SQLite Joey Bernard is a true renaissance man. He splits his time between building furniture, helping researchers with scienti?c computing problems and writing Android apps. Why Python? It?s the of?cial language of the Raspberry Pi. Read the docs at python.org/doc In previous issues, we have looked at how to use data that is stored in ?les using regular ?le I/O. From here, we moved on to looking at how to use pandas to work with more structured data, especially in scienti?c work. But, what do you do when you have data that needs that go beyond these tools, especially in non-scienti?c domains? This is where you likely need to start looking at using a database to manage information in a better way. This month, we will start by looking at some of the lighter options available to work with simple databases. This gives you a connection object that enables interactions with the database stored in the example.db ?le in the current directory. If it doesn?t already exist, the sqlite3 module will create a new database ?le. If you only require a temporary database that needs to live for the duration of your program run, you can give the connection method the special ?lename ?:memory:? to create a database stored solely in RAM. Now that you have a database, what can you do with it? The ?rst step is to create a cursor object for the database to handle SQL statements being applied to the database. You can do so with my_cursor = my_conn.cursor(). The ?rst database thing you will need to do is to create tables to store your data. As an example, the following code creates a small table to store names and phone numbers. "All of the SQLite code is part of your code" In terms of lightweight databases, SQLite is the de facto standard in most environments. It comes as a C library that provides access to a ?lebacked database that is stored on the local disk. One huge advantage is that it does not need to run a server process to manage the database. All of the code is actually part of your code. The query language used is a variant of standard SQL. This means that you can start your project using SQLite, and then be able to move to a larger database system with minimal changes to your code. There is a port to Python available in the module ?sqlite3? which supports all of the functionality. Because it is the standard for really lightweight database functionality, it is included as part of the standard Python library. So you should have it available wherever you have Python installed. The very ?rst step is to create a connection object that starts up the SQLite infrastructure: import sqlite3 78 my_conn = sqlite3. connection('example.db') my_cursor.execute('''CREATE TABLE phone (name text, phone_num text)''') You have to include the data type for each column of the table. SQLite natively supports SQL data types BLOB, TEXT, REAL, INTEGER and NULL. These map to the Python data types byte, str, ?oat, int and None. The execute method runs any single SQL statement that you need to have run against the database. These statements are not committed to the ?le store for the database, however. In order to have the results actually written out, you need to run my_conn.commit(). Note that this method is part of the connection object, not the cursor object. If you have a different thread also using the same SQLite database ?le, it won?t see any changes until a commit is called. This means that you can use the rollback() method to undo any changes, back to the last time commit() was called. This allows you to have you a rudimentary form of transactions, similar to the functionality of larger relational databases. Now that we have a table, we should start populating it with data. The simplest way to do this is to use a direct INSERT statement, e.g.: my_cursor.execute("INSERT INTO phone VALUES ('Joey Bernard', '555-5555')") While this is okay for hard-coded values, you?ll probably have data coming from the user that needs to be entered into the database. In these cases, you should always check this input and sanitise it so there?s no code that can be used for an SQL injection attack. You can do this, then do string manipulation to create the complete SQL statement before calling the execute method. The other option available is to use an SQL statement that contains placeholders that can be replaced with the values stored in variables. This makes the validation of the input data a bit easier to handle. The above example would then look like: my_name = 'Joey Bernard' my_number = '555-5555' my_cursor.execute("INSERT INTO phone VALUES (?,?)", (my_name,my_number)) The values to be used in the SQL statement are provided within a tuple. If you have a larger amount of data that needs to be handled in one go, you can use the executemany() What if SQLite isn?t light enough? function, available in the cursor object. In this case, the SQL statement is structured the same as above. The second parameter is any kind of iterable object that can be used to get a sequence of values. This means that you could write a generator function if your data can be processed that way. It is another tool available to automate your data management issues. Now that we have some data in the database, how can we pull it back out and work with it? The basic SQL statement that is used is the SELECT statement. You can use the following statement to get my phone number. my_cursor.execute("SELECT phone_num FROM phone WHERE name=:who", {"who":'Joey Bernard'}) print(my_cursor.fetchone()) As you can see, you need to call some kind of fetching method in order to get your actual results back. The fetchone() method returns the next returned value from the list of returned values. When you reach the bottom of the list, it will return None. If you want to process returned values in blocks, you can use the cursor method fetchmany(size), where size is how many items to return bundled within a list. When this method runs out of items to return, it sends back an empty list. If you want to get the full collection of all items that matched your SELECT statement, you can use the fetchall() method to get a list of the entire collection. You do need to remember that any of the methods that return multiple values still start wherever the cursor currently is, not from the beginning of the returned collection. Sometimes, you may need to add some processing functionality to the database. In these cases, you can actually create a function that can be used from within other SQL statements. For example, you could create a database function that returns the sine of some value. import math my_conn.create_function("sin", 1, math.sin) cursor2 = my_conn.cursor() cursor2.execute("SELECT sin(?)", (42,)) print(cursor2.fetchone()) There is a special class of database functions, called aggregators, that you may wish to create, too. These take a series of values and apply an aggregation function, like summing, over all of them. You can use the create_aggregate() method to register a class to act as the new aggregator. The class needs to provide a step() method to do the aggregation calculation and a ??λ?????? method that returns the ?nal result. One last item you may want to be able to do is to have larger blocks of SQL statements run against your data. In this case, you will want to use the cursor object?s executescript() method. This method takes a string object that contains an entire script and runs it as a single block. One important difference here is that a commit() call is made just before your script is run. If you need to be able to do rollbacks, this is an important caveat to keep in mind. When you start to have more complicated queries, you may need to track where your results came from. The description property of the cursor object returns the column names for the last executed query. When you are all done, you should always call the close() method of the connection object. But, be aware that a commit is not done automatically. You will need to call it yourself before closing. This ensures that all of your transactions are ?ushed out to disk and the database is in a correct state. Now you can add more robust data management to your code. Sometimes, even SQLite may not be lightweight enough, depending on your speci?c requirements. In these cases, you do have another option. There is a very old ?le-backed database from the earliest days of UNIX, called dbm. Dbm databases store data as a set of key-value pairs within a ?le on the ?le system. To start, you will need to open a database with code like that given below. import dbm db = dbm.open('example.db', 'c') This opens the database in the ?le example.db, or creates it if it doesn?t already exist. You can insert a value to a given key, or get a value based on a key. Below is an example of storing a name/phone number pair. db['Joey Bernard'] = '555-5555' When you do the query, you need to remember that everything is stored as byte strings, so you will need to use those to get values. my_number = db.get(b'Joey Bernard') There are two more advanced variants available, gdbm and ndbm. They each add some further functionality above and beyond that provided by the basic dbm implementation. One important thing to note is that the ?le formats for the different variants are not compatible. So if you create a database with gdbm, you will not be able to read it with ndbm. There is a function, named whichdb(), that will take a ?lename and try to ?gure out which type of dbm ?le it is. Gdbm has methods to easily allow you to traverse the entire database. You start by using the ????????? method to get the ?rst key in the database. You can then travel through the entire database by using the method nextkey(key). Gdbm also provides a method, named reorganize(), which can be used to collapse a database ?le after a number of deletions. Because dbm, and its variants, store data as sets of key/value pairs, it maps quite naturally to the concepts around dictionaries. You can use the same syntax, including the ?in? keyword, from dictionaries when you work with any of the dbm variants. These modules allow you to have data stores that you can use to store simpler data structures within your own programs. www.linuxuser.co.uk 79 ;OLZV\YJLMVY[LJOI\`PUNHK]PJL techradar.com 81 Group test | 86 Hardware | 88 Distro | 90 Free software Antergos Chakra GNU/Linux Manjaro Linux RevengeOS GROUP TEST Arch-based distributions Experience the bene?ts of the venerable Arch Linux in the comforts of a desktop distribution that takes the pain out of that installation Antergos A multilingual distribution that?s proud of the fact that it?s designed with simplicity in mind. Antergos ships with custom artwork and claims it provides a ready-to-use system that doesn?t require any additional steps once it?s been installed. https://antergos.com Chakra GNU/Linux One of the two most popular Arch-based distributions designed for desktop users, Chakra uses a half-rolling release cycle. The distro marks certain packages as core that only receive updates to ?x security issues while other apps are on a rolling-release model. https://chakralinux.org Manjaro Linux RevengeOS The other popular Arch-based desktop distribution, Manjaro, uses its own set of repositories. The distribution is available with three different desktops ? Xfce, KDE and Gnome ? and the latest release has been announced as the project?s last to offer support for 32-bit hardware. https://manjaro.org The oddly-named distribution is all about choice ? starting from its six desktop environments to the default apps. RevengeOS? goal is to provide an easy-toinstall Arch distribution while preserving the power and customisation offered by its Arch Linux base. http://revengeos.weebly.com www.linuxuser.co.uk 81 Review Arch-based distributions Antergos Chakra Linux Makes good use of Arch?s large repos to offer seven desktops A perfect rendition of KDE with an app selection to match Q You can enable the Arch User Repository (AUR), in addition to Antergos? own repo, from the very useful Cnchi installer Q Chakra has several custom repositories, including a communitymaintained repository that?s inspired by the Arch User Repository Installation Installation The home-brewed, multilingual Cnchi installer does a good job of anchoring the distro. It allows you to choose from the seven supported desktops and can also install an Arch-like, CLI-only base. The partitioning step is intuitive enough for new users and also helps advanced ones use ZFS, setup LVM and home on a separate partition. Chakra uses the distribution-independent Calamares installer. It?s intuitive and can be navigated with ease. Like Cnchi, you can ask Calamares to automatically partition the disk or let you modify partitions manually. However, Calamares doesn?t offer as many options as Cnchi installer. Pre-installed apps Pre-installed apps Antergos has the usual slew of productivity apps. By default, the distro uses the Chromium web browser but you can install Firefox during installation where you also get options to pull in LibreOf?ce and Steam. The default Gnome desktop also includes Gnome tweaks and dconf editor to tweak the Gnome desktop. Chakra expressly bills itself as a KDE-centric distribution, which is why it includes a whole gamut of KDE apps including the complete Calligra of?ce suite. It also includes other Qt apps such as the Bomi media player and the lightweight QupZilla browser in place of a more feature-rich browser like Firefox. Usability Usability Besides the project?s Cnchi installer, there are no custom tools of the project. The distro allows you choose between cinnamon, deepin, gnome, KDE, Mate, Openbox and Xfce. All desktops come with their own con?guration tools and they all have been tweaked to look very pleasing with a wide selection of desktop wallpapers, along with the Numix Frost theme. Chakra uses Octopi for package management and for keeping your installation updated. It also includes a cache cleaner and a repository editor. In terms of overall usability, KDE has always had a tendency to overwhelm the new user, which makes Chakra best suited for those who are already familiar and comfortable with the desktop. Help & support Help & support The project has very active multilingual forum boards with separate sections for installation, resolving newbie queries, applications and desktop environments, pacman & package upgrade issues, etc. There are several useful articles in the well categorised wiki and you can also seek advice on the project?s IRC channel. The project has recently overhauled its support infrastructure and the developers now engage with the users via the community section on the website. You can still ?nd the link to the old wiki that has several useful articles to help install and manage the distribution, while others only have skeletal content. Overall Overall Antergos is an aesthetically pleasing distribution. Its Cnchi installer does a wonderful job and helps users customise various aspects of the installation including app selection. 82 8 An Arch-based, 64-bit-only rolling release distro, Chakra?s objective is to give its users the ultimate KDE experience. It?s an excellent desktop distro for those whose demands align with its goals. 7 Manjaro Linux RevengeOS Helps all users leverage the power of Arch with its powerful custom tools A lightweight distribution with tools to help you choose and customise Q With MSM you can install proprietary drivers for connected hardware and even switch between available kernels with a single click QWith the distros? Software Installation Tool you can install popular productivity and multimedia apps with a single click Installation Installation Manjaro is available in three of?cially supported ?avours and while the project doesn?t recommend one over another, we?ve used the Xfce edition that is listed ?rst on the download page. All editions use a customised Calamares installer that enables you encrypt the partition in which you plan to install the distribution. RevengeOS uses its own home-brewed Nemesis installer. It offers three types of installs ? Normal, OEM, and StationX. The installer is fairly intuitive to navigate and allows you choose between six desktop environments. The partitioning step offers an automatic installation option but just ?res up Gparted for manual partitioning. Pre-installed apps Pre-installed apps Manjaro includes all the usual mainstream popular apps such as the LibreOf?ce suite, GIMP, VLC, Firefox, Thunderbird, Steam client, etc. The remaining space is stuffed with handy utilities from the Xfce stable. If you need more apps from the repos, the Xfce edition uses Pamac for package management and handling updates. During installation you get options to install certain useful software such as LibreOf?ce, Wine, Steam client, etc. Besides these, the distro contains the usual collection of apps that ship with the desktop you?ve installed. We opted for its customised OBR Openbox desktop that installs several Xfce tools and some extras like the Gufw ?rewall. Usability Usability Manjaro is very usable since it bundles the regular apps instead of esoteric alternatives. The Xfce desktop follows the conventional desktop metaphor and will suit a large number of potential users. Another plug is the project?s custom tools ? especially the Manjaro Settings Manager ? that allow users really take advantage of the Arch base. The distro boots to a welcome screen that points to scripts to remove the pre-installed virtualbox modules and another to install proprietary Nvidia drivers. There?s also a custom control centre for accessing various custom tools for managing all aspects of the system, from changing wallpapers and docks to installing proprietary codecs and switching kernels. Help & support Help & support Manjaro trumps the lot with a channel on Vimeo with over two dozen videos on various aspects of the distribution and its development. Their wiki is fairly detailed and well categorised, and the project hosts multilingual forums and IRC channels. The distro also bundles a detailed PDF user guide. The welcome screen has links to join the project?s Google+ community page and another to the distro?s fairly active forum boards. Here you?ll ?nd boards for posting problems regarding installation and any issues with the apps. The website also has a wiki with some articles that you may ?nd useful. Overall Overall Manjaro?s developers have gone to great lengths to help users experience the power of Arch. With its custom tools and bundled apps, the ready-to-use desktop scores highly in both form and function. 9 Over?owing with custom tools and scripts, RevengeOS is a well crafted distribution that does a wonderful job of handholding new users and helping them get to grips with Arch. 8 www.linuxuser.co.uk 83 Review Arch-based distributions In brief: compare and contrast our verdicts Antergos Installation Its installer helps you pick a desktop and has smart partitioning options. Preinstalled apps You can use the installer to grab apps like Firefox, LibreOf?ce & Steam. Usability All of the supported desktops have been tweaked to look good and are very usable. Help & Support Offers enough options to help you ?nd your way around any possible issues. Overall A smart-looking distro with an installer that helps customise the install. Chakra GNU/Linux 8 Uses the Calamares installer that?s not as functional as Cnchi but works well. 8 Takes pride in being a KDE-only distro and packs in all sorts of Qt apps. 8 The inclusion of Qtonly apps over some mainstream ones can be an issue for some. 8 It?s new but the recently overhauled support section will ?nd you a solution. 8 The ultimate Archbased distro for users looking for a pure KDE experience. Manjaro Linux 7 The Calamares installer offers users the option of encrypting the installation partition. 8 Includes all of the popular apps you?d expect in a modern desktop distro. 7 The custom tools and the bouquet of apps make it easy to use and tweak. 7 The usual textbased options are complemented by videos on the Vimeo channel. 7 Allows users of all skill levels to experience the power and ?exibility of Arch. RevengeOS 8 Its Nemesis installer is very usable and supports three types of installs. 8 9 The installer helps you to pick the desktop as well as popular, everyday apps. 8 9 You can mould the installation as per your needs with its custom control centre. 9 9 You get a Google+ page and a couple of boards on the forum to post your queries. 7 9 Essentially a one-man show that includes a great many tools to shape the installation. 8 AND THE WINNER IS? Manjaro Linux After Ubuntu, Arch Linux is one of the most popular projects that?s used as a base for creating all kinds of niche as well as general purpose distributions. In this group test, we?ve included the ones that focus on helping new and inexperienced users uncover the power of Arch. These distros help save them the time and effort investment that?s typically required in deploying an Arch install. Chakra ruled itself out of contention for its affection towards KDE users, which, with its myriad of options, isn?t always the best environment for inexperienced users. RevengeOS and its gamut of custom tools make for a pleasant experience but the distro is a one-man show without a ?xed release schedule. Plus if you don?t like the default theme, tweaking the visuals will take some time and effort that?s best spent elsewhere. In the end, the real contest ? and a close one at that ? was between Antergos and Manjaro. A big plus for Antergos is that it offers the option of multiple desktops during installation. It?s also visually appealing but apart from being a Cnchi installer, there are no other custom tools. 84 Q Use the CLI Manjaro architect tool to install a customised build with your choice of kernel, drivers etc Manjaro, on the other hand, includes the very useful Manjaro Settings Manager that helps users take advantage of the Arch ecosystem without getting into the nittygritties. It?s also chock-full of apps and can be used straight out of the box. The project has its own dedicated software repositories that deliver thoroughly tested and stable software that works well in conjunction with Arch and AUR repositories and can be used for bleeding-edge software. Finally, Manjaro remains one of the very few independently developed projects that is still on the market for both 32-bit and 64bit architectures. Mayank Sharma Not your average technology website EXPLORE NEW WORLDS OF TECHNOLOGY GADGETS, SCIENCE, DESIGN AND MORE Fascinating reports from the bleeding edge of tech Innovations, culture and geek culture explored Join the UK?s leading online tech community www.gizmodo.co.uk twitter.com/GizmodoUK facebook.com/GizmodoUK iStorage DiskAshur Pro 2 Review HARDWARE iStorage diskAshur Pro 2 Price From £195 for 500GB (reviewed £489, 5TB) Website https://istorage-uk.com Specs Capacity: 500GB-5TB Data transfer speed: Up to 148MBps (Read),140MBps (Write) Power supply: USB bus-powered Interface: USB 3.1?up to 5Gbps Hardware data encryption: RealTime Military Grade AES-XTS 256-bit Full-Disk Hardware Encryption Warranty: 2 years Dimensions (W, D, H): 124mmx84mmx20 mm (500GB,1/2TB), 124mmx84mmx28mm (3/4/5TB) Weight: 500GB/1/2TB 225g, 3/4/5TB 331g 86 An expensive portable hard drive on the face of it, but good security has never been cheap For those who carry sensitive information around with them on a daily basis, there?s an everpresent concern with losing the device carrying that precious data (or worse, having it stolen). This is concern addressed by the new iStorage diskAshur Pro2 external HD, a compact storage device designed to work with secure data without the need to install software on all the systems it will meet. The Pro2 retails at £489 ($651), comes in 500GB, 1TB, 2TB, 3TB, 4TB and 5TB capacities, and it was the latter which the ?rm supplied to us for review. The Pro2 is expensive as 5TB external drives go, being quadruple what Seagate asks for its Backup Plus 5TB drive, and more than double LaCie?s rugged Thunderbolt 5TB models. But after removing the drive from the packaging, we realised why iStorage asks so much for it, because this is, without doubt, one of the most glorious pieces of product engineering we?ve had the pleasure to handle. The upper and lower surfaces are cool-to-the-touch metal, and the waistband is softtextured rubber. The drive is just 84mm wide, 124mm long and 20mm deep, dictating that this uses 2.5-inch drives internally to provide 5TB of capacity. As if to underline how much of the cost goes into the engineering and not the capacity, the 2TB model is £329 ($437 approx), only £160 ($212 approx) cheaper. The unit has numerous hardware safeguards to defend against external tampering, bypass attacks and fault injections From a design perspective, two features make this drive special: the ?rst is the built-in USB cable. The cable is only 12cm long when unclipped, but that?s enough to attach it to a laptop or desktop PC. The other standout feature is the built-in numeric pad. This is an integral feature of the security mechanism iStorage has implemented. In addition to the numbers, there are a few special keys for operating the unit when it?s attached to a computer. Above the numeric pad are three LEDs that con?rm the locked condition, and also show drive activity. At the heart of this design is a secure microprocessor (Common Criteria EAL4+ ready) that handles the encryption of the device. This, in theory, means that if the bare drive is extracted from its case, an attacker is no closer to getting to the data. A data thief will need the 7-15 digit numeric password created when the Pro2 was last con?gured. If you fail to enter the correct code enough times then this will result in the drive deleting the encryption key, rendering the contents beyond reach forever, unless you know people in the security services who can crack AES-XTS 256-bit. In addition, the unit also has numerous hardware safeguards to defend against external tampering, bypass attacks and fault injections. Should it detect any attempt to get into the case or tinker with USB, it will trigger a deadlock frozen state, at which point further assault is pointless. Devices with a numeric pad like this usually come with a PC application that you?ll need to install, but the Pro2 is fully self-contained. That allows it to work as well with Linux as many other OSes it supports, such as Android, Chrome, MacOS and Window. You can format it to whatever ?lesystem you use?even one you?ve created yourself. The unit comes with a default admin PIN number de?ned, and you can change that directly using the pad. But the most Bond-esque PIN code you can de?ne is the one that initiates a ?self-destruct? sequence, which performs an internal crypto-wipe where all the PINs and data are erased, and the drive must be reformatted before it can be used again. For the majority of people, the Pro2 has enough in the way of protection. With 145.5MBps reads and 144.8MBps writes, the spinning disk inside the Pro2 has some intent about it. While an SSD would be quicker (and iStorage provides models with those inside, too), those performance levels are good enough, and about as rapid as a PC with hard disk-based storage is likely to be. The unit is also IP56 certi?ed, making it water and dust-resistant, though not waterproof. An extra touch in terms of the physical protection is that the keys on the pad are coated in epoxy. The coating makes it harder to work out what keys are being used on a regular basis. Mark Pickavance Pros Beautiful construction and offers genuine data security. Being self-contained, it will work with any Linux distro Cons You get what you pay for, so it?s expensive and also potentially a little daunting to set up Summary A genuinely secure storage device that?s built to handle physical abuse and nefarious decryption ? but it comes at a premium price. The overall combination of a well-considered security model which is also a superbly engineered device is an alluring one. 9 www.linuxuser.co.uk 87 Review Pop!_OS 17.10 Above Pop is available only for 64-bit architecture with separate releases for Intel/AMD and Nvidia hardware DISTRO Pop!_OS 17.10 Caught out by the dramatic changes in Ubuntu, System76 decides to take matters into its own hands RAM 2GB Storage 20GB Specs CPU: 64-bit Intel or AMD processor Graphics: AMD or Nvidia hardware Available from: https:// system76.com/pop 88 System76 is one of the few retailers to sell computers with 100 per cent Linux compatible hardware pre-loaded with Ubuntu. The desktop may not be Canonical?s core business, but it is at System76?the end of the Unity project affected 91 per cent of its business. In response, it put together a distribution of its own to offer customers a user experience in line with the company?s hardware. The company went distro hopping and settled on the latest Ubuntu with GNOME Shell as the base for its custom distro christened Pop!_OS. Instead of focusing its efforts on mainstream desktop users, System76 is designing a distro geared towards creators, developers and makers: ?If you?re in software engineering, scienti?c computing, robotics, AI or IOT, we?re building Pop!_OS for you,? says CEO, Carl Richell in his blog post announcing the beta release. ?We?ll build tools to ease managing your dev environment. We?ll make sure CUDA is easily available and working.? System76 raised our expectations with its ?Imagine? marketing (https://system76.com/pop) that promises ?the most productive and gorgeous platform for developing your next creation?, even if it adds ?we?re just getting started?. Speaking to Richell on the ?rst full release, he explained that the company?s intention is to establish the ?guiding principles of the distribution, develop the aesthetic, build infrastructure, testing and quality procedures, start a community, documentation, web pages, production (shipping on laptops and desktops at scale) and of course, the release.? Without this background, the ?rst release seems bland and minimalistic given the objectives on its website. ?In four months we?ve laid the infrastructure and direction of our OS. The next six months are Above Sure, Pop is pitched at users who might not be aware of Gnome, but not mentioning the desktop even once might alienate those in the know As a good open source supporter, System76 is also improving upstream code executing on our vision?, explains Carl. We feel it?s important to bear this in mind when assessing the distro, but we can?t review what isn?t there yet. On the software side, the distro is pretty standard desktop fare, with LibreOf?ce, Firefox and Gnome?s default bouquet of applications for viewing images and videos. Compared to Ubuntu, Pop uses Geary for email instead of Thunderbird and doesn?t include the default Ubuntugames, the pesky Amazon integration, Cheese, Transmission, Rhythmbox, and Shotwell. Carl says that in the company?s experience these missing apps aren?t widely used by customers. Pop?s installation is based on Ubuntu?s Ubiquity installer. The only real difference apart from cosmetic ones is that there?s no initial user creation step. This has been moved to a post-install ?rstboot wizard. Pop?s GNOME desktop has been tweaked to deliver a user experience that best suits its customers. GNOME extensions are enabled that display workspaces whenever you bring up the Activities Overview, and another that adds a Suspend button in the Power menu. To protect your privacy, Pop also doesn?t display noti?cations on the lock screen. The System76 UI team has spent considerable time getting the visual details right. The ?at Pop GTK theme based on the Materia GTK+ theme with matching cursor and icons are pleasing. As a good open source supporter, System76 is also improving upstream code. Carl tells us that it?s improved GNOME?s half-tiling window focusing and contributed patches to ensure HiDPI works properly. The one GNOME tweak that made things dif?cult for us were the remapped keyboard shortcuts. Carl tells us that the shortcuts have been worked out with input from software developers, but the tweaks necessitated a visit to the keyboard settings section to reassign them to the familiar GNOME values. Another highlight is the Pop!_Shop app store, which is based on code from elementary OS?s AppCenter. While the distro uses the same repositories as Ubuntu 17.10, System76 handpicks the listed apps. This is why you can?t install apps such as Thunderbird, Evolution, Chromium from the Pop!_Shop. However, you can fetch them using APT. Another application that Pop has borrowed from the elementary OS project is Eddy, the DEB package installer. Carl tells us System76 are considering creating ?application suites? that will help users fetch multiple apps relevant to a particular task. Mayank Sharma Pros An aesthetically pleasing GNOME rendition with useful extensions enabled by default and an intuitive app store. Cons A minimalist distro with a default application selection that so far fails to meet its grandiose objectives. Summary A good-looking distro that doesn?t yet match what it promises on the of?cial website. But much of the work in this ?rst release is in the background and lays the groundwork for future developments that?ll help push the distro as the perfect platform for developers. 7 www.linuxuser.co.uk 89 Review Fresh free & open source software CODE EDITOR CudaText 1.23.0 A feature-rich editor for writing code Linux has no dearth of advanced text editors that moonlight as lightweight IDEs as well and we ran a group test of some of the best a few issues ago (LU&D182, Pg 81). CudaText is one such crossplatform editor that?s primarily designed to write code, but can double up as an advanced text editor. The editor has all the usual coding conveniences, like syntax highlight for several programming languages, including C, C++, Java, JavaScript, HTML, CSS, Python, and more. You also get code completion for some languages like HTML and CSS, code folding, the ability to search and replace with regex, as well multi-caret editing and multi-selection. You can extend the functionality by installing additional plugins that are written in Python. The project releases precompiled binaries for Debian-based distros. On others you?ll have to compile it manually by following the instructions on the wiki. It has a single document that explains all aspects of the app, and is a must-read to get to grips with all the functionalities of the app. If you?ve worked with advanced text/code editors before, you?ll have no issues navigating CudaText?s interface. Above The app has colour themes for the interface. Each has a matching scheme for coding syntax Pros Includes all the common coding conveniences and can be extended with plugins. Cons You?ll have to edit its con?guration ?le to hook it up with the Python interpreter. Great for? Editing code without an IDE. http://uvviewsoft.com/ cudatext/ DIVE LOG TOOL Subsurface 4.7.1 Log and analyse all your scuba dives with ease Linus Torvalds likes to track whatever he does. He wrote Git to keep track of the kernel development and Subsurface to log his dives. Torvalds likes to don the wetsuit and plunge underwater whenever he isn?t busy hacking away at the Linux kernel. He couldn?t ?nd a good app to log his dives, so naturally he wrote one himself. Simply put, Subsurface helps keep track of scuba dives. You can use it to import data from one of the supported dive computers, usually via USB. Once your data is imported, you can view and edit dive details from the intuitive user interface. The app also enables you to log dives manually. The app shows a dive pro?le graph that plots dives as a line graph. 90 It shows ascent and descent speeds along with the depths. Certain events, such as the begin of a decompression, are also marked. The app also records details about the diving equipment, and can calculate various stats about multiple dives. There?s a detailed user manual to help acquaint users, as well as a video tutorial. The latest version sports some user interface changes, like a new map widget. Support for importing dive data from Shearwater desktop, DL7, Datatrak and in other third-party formats has also been improved, and the version has experimental support for Bluetooth LE dive computers. The project lists binaries on its Downloads page for several popular distros, and there?s an AppImage that?ll work on any Linux distro. Pros Very easy to install and works with a large number of dive computers. Cons You?ll have to read through its documentation to discover all its features. Great for? Analysis of dive data. https://subsurfacedivelog.org DESKTOP ENVIRONMENT LXQt 0. 1 2.0 Bring old workhorses to life with this lightweight desktop environment The Lightweight Qt Desktop Environment?called LXQt for short?will draw a graphical user interface without consuming too many resources. The desktop environment is a mix of GTK-based lightweight desktop, LXDE, and RazorQt, which was an equally lightweight?but less mature?desktop that used the Qt toolkit. The recent releases of the mainstream desktop environments such as Gnome and KDE have put them out of the reach of low-spec machines, which is why many popular distros have a LXQt-based ?avour in their arsenal of releases. LXQt is also popular with users of newer more powerful machines, as it helps free the resources for more CPU-intensive tasks such as video processing. Pros The latest version of the desktop includes better support for HiDPI displays. The new release ships with a new Open/Save File dialog, and includes support for icon themes that use the FollowsColorScheme KDE extension to the XDG icon themes standard. Behind the scenes, the developers have also improved the shutdown/reboot process by shutting down all LXQt components before allowing systemd to do its job. There have been some important architectural changes too. The lxqt-common package has been dropped, and its components split into several other packages, such as the newly introduced LXQt-themes package. The project?s website has a list of distros that ship the LXQt desktop either in a spin or via their repositories. A fully functional and smartlooking desktop that doesn?t consume too many resources. Cons Doesn?t include the bells and whistles you get with the mainstream desktops environments. Great for? Busy and old PCs. https://github.com/pixlra/ playuver IMAGE CONVERTER Converseen 0.9.6.2 Automate repetitive image processing tasks If you?ve worked with images, either professional or sorting through vacation clicks, you know a lot of image processing work is monotonous?repeatedly converting, resizing and rotating images to make them suitable for print or passing around. Although you can use virtually any image viewing or editing app, from DigiKam to GIMP, for this task, you?ll save yourself hassle by employing a dedicated batch conversion tool like Converseen. Converseen is a straightforward frontend to the command-line conversion utilities in ImageMagick. You can use the app to convert images to and from over 100 formats, rotate and ?ip them, change their dimensions and rename them in a fraction of the time it would take to perform these tasks manually. You?ll ?nd installable binaries for Ubuntu, Fedora and OpenSUSE on the project?s website, along with simple installation instructions. The work?ow is pretty straightforward?click the Add Images button to select any number of images you want to convert. Then scroll down the Action panel on the left to specify how you want to manipulate the image, and hit the Convert button to begin the process. Above Converseen exposes only a fraction of the image manipulation tricks you can do with ImageMagick Pros Straightforward interface that helps automate mundane image editing tasks Cons Great for? Exposes a very limited subset of the power of the CLI tools it?s based on. Resizing, rescaling and rotating. http://converseen.fasterland. net www.linuxuser.co.uk 91 Web Hosting Get your listing in our directory To advertise here, contact Chris chris.mitchell@futurenet.com | +44 01225 68 7832 (ext. 7832) RECOMMENDED Hosting listings Featured host: Use our intuitive Control Panel to manage your domain name www.thenames.co.uk 0370 321 2027 About us Part of a hosting brand started in 1999, we?re well established, UK based, independent and our mission is simple ? ensure your web presence ?just works?. We offer great-value domain names, cPanel web hosting, SSL certi?cates, business email, WordPress hosting, cloud and VPS. What we offer � Free email accounts with fraud, spam and virus protection. � Free DNS management. � Easy-to-use Control Panel. � Free email forwards ? automatically redirect your email to existing accounts. � Domain theft protection to prevent it being transferred out accidentally or without your permission. � Easy-to-use bulk tools to help you register, renew, transfer and make other changes to several domain names in a single step. � Free domain forwarding to point your domain name to another website. 5 Tips from the pros 01 Optimise your website images When uploading your website to the internet, make sure all of your images are optimised for the web! Try using jpegmini.com software; or if using WordPress, install the EWWW Image Optimizer plugin. 02 Host your website in the UK Make sure your website is hosted in the UK, not just for legal reasons! If your server is located overseas, you may be missing out on search engine rankings on google.co.uk ? you can check where your site is based on www.check-host.net. 03 Do you make regular backups? How would it affect your business if you lost your website today? It?s vital to always make your own backups; even if 92 your host offers you a backup solution, it?s important to take responsibility for your own data and protect it. 04 Trying to rank on Google? Google made some changes in 2015. If you?re struggling to rank on Google, make sure that your website is mobile-responsive! Plus, Google now prefers secure (HTTPS) websites! Contact your host to set up and force HTTPS on your website. 05 Testimonials David Brewer ?I bought an SSL certi?cate. Purchasing is painless, and only takes a few minutes. My dif?culty is installing the certi?cate, which is something I can never do. However, I simply raise a trouble ticket and the support team are quickly on the case. Within ten minutes I hear from the certi?cate signing authority, and approve. The support team then installed the certi?cate for me.? Tracy Hops ?We have several servers from TheNames and the network connectivity is top-notch ? great uptime and speed is never an issue. Tech support is knowledge and quick in replying ? which is a bonus. We would highly recommend TheNames. ? Avoid cheap hosting We?re sure you?ve seen those TV adverts for domain and hosting for £1! Think about the logic? for £1, how many J Edwards ?After trying out lots of other hosting companies, you clients will be jam-packed onto that seem to have the best customer service by a long way, server? Surely they would use cheap £20 and all the features I need. Shared hosting is very fast, drives rather than £1k+ enterprise SSDs! and the control panel is comprehensive?? Remember: you do get what you pay for! SSD web hosting Supreme hosting www.bargainhost.co.uk 0843 289 2681 www.cwcs.co.uk 0800 1 777 000 Since 2001, Bargain Host has campaigned to offer the lowest possible priced hosting in the UK. It has achieved this goal successfully and built up a large client database which includes many repeat customers. It has also won several awards for providing an outstanding hosting service. CWCS Managed Hosting is the UK?s leading hosting specialist. It offers a fully comprehensive range of hosting products, services and support. Its highly trained staff are not only hosting experts, they?re also committed to delivering a great customer experience and passionate about what they do. � Colocation hosting � VPS � 100% Network uptime Value hosting elastichosts.co.uk 02071 838250 � Shared hosting � Cloud servers � Domain names Enterprise hosting: Value Linux hosting www.2020media.com | 0800 035 6364 WordPress comes pre-installed for new users or with free managed migration. The managed WordPress service is completely free for the ?rst year. We are known for our ?Knowledgeable and excellent service? and we serve agencies, designers, developers and small businesses across the UK. ElasticHosts offers simple, ?exible and cost-effective cloud services with high performance, availability and scalability for businesses worldwide. Its team of engineers provide excellent support around the clock over the phone, email and ticketing system. www.hostpapa.co.uk 0800 051 7126 HostPapa is an award-winning web hosting service and a leader in green hosting. It offers one of the most fully featured hosting packages on the market, along with 24/7 customer support, learning resources, as well as outstanding reliability. � Website builder � Budget prices � Unlimited databases Linux hosting is a great solution for home users, business users and web designers looking for cost-effective and powerful hosting. Whether you are building a single-page portfolio, or you are running a database-driven ecommerce website, there is a Linux hosting solution for you. � Student hosting deals � Site designer � Domain names � Cloud servers on any OS � Linux OS containers � World-class 24/7 support Small business host patchman-hosting.co.uk 01642 424 237 Fast, reliable hosting Budget hosting: www.hetzner.de/us | +49 (0)9831 5050 Hetzner Online is a professional web hosting provider and experienced data centre operator. Since 1997 the company has provided private and business clients with high-performance hosting products, as well as the necessary infrastructure for the ef?cient operation of websites. A combination of stable technology, attractive pricing and ?exible support and services has enabled Hetzner Online to continuously strengthen its market position both nationally and internationally. � Dedicated and shared hosting � Colocation racks � Internet domains and SSL certi?cates � Storage boxes www.bytemark.co.uk 01904 890 890 Founded in 2002, Bytemark are ?the UK experts in cloud & dedicated hosting?. Their manifesto includes in-house expertise, transparent pricing, free software support, keeping promises made by support staff and top-quality hosting hardware at fair prices. � Managed hosting � UK cloud hosting � Linux hosting www.linuxuser.co.uk 93 Get your free resources Download the best distros, essential FOSS and all our tutorial project files from your FileSilo account WHAT IS IT? Every time you see this symbol in the magazine, there is free online content that's waiting to be unlocked on FileSilo. WHY REGISTER? � Secure and safe online access, from anywhere � Free access for every reader, print and digital � Download only the ?les you want, when you want � All your gifts, from all your issues, all in one place 1. UNLOCK YOUR CONTENT Go to www.?lesilo.co.uk/linuxuser and follow the instructions on screen to create an account with our secure FileSilo system. When your issue arrives or you download your digital edition, log into your account and unlock individual issues by answering a simple question based on the pages of the magazine for instant access to the extras. Simple! 2. ENJOY THE RESOURCES You can access FileSilo on any computer, tablet or smartphone device using any popular browser. However, we recommend that you use a computer to download content, as you may not be able to download ?les to other devices. If you have any problems with accessing content on FileSilo, take a look at the FAQs online or email our team at ?lesilohelp@futurenet.com. Free for digital readers too! Read on your tablet, download on your computer 94 Log in to www.filesilo.co.uk/linuxuser Subscribeandgetinstantaccess Get access to our entire library of resources with a moneysaving subscription to the magazine ? subscribe today! Thismonthfind... DISTROS In the FileSilo this month you will ?nd Antergos 17.11, an easy-to-install distro powered by Arch Linux, along with Xubuntu 17.10 an optimised Ubuntu distro. SOFTWARE As well as two full distros, we've bundled together a Rescue and Repair kit that includes two popular live distros, SystemRescueCd and Rescatux, that will help you bring a sick system back to life. TUTORIAL CODE This month we've got the skeleton project for MQTT, Python code for Arduino and a TAR for the Java series and more! Subscribe & save! See all the details on how to subscribe on page 30 Short story FOLLOW US NEXT ISSUE ON SALE 11 JANUARY Stephen Oram YOUR FREE DISC | Microsoft loves Linux? | SBC Round-up Facebook: Twitter: facebook.com/LinuxUserUK @linuxusermag NEAR-FUTURE FICTION Disjointed e stood naked in front of the full-length mirror, ?exing his biceps. The mirror ?ashed an amber warning, reminding him to stand st scanned his organs, blood, bones and skin. It would evaluate his health and adjust the multitude of enhancement implants scattered throughout his body, ?ne-tuning them as it went to maximise his physical and mental performance. This daily routine made him feel trapped and cornered, as if the mirror was a docking bay that he couldn?t stay away from for more than twenty-four hours. He wanted to run into the sea or drink himself silly. He wanted to go off-grid and wander wherever he liked and for as long as he liked. The pressure to break free had been building for a while, to such an extent that he doubted whether he could make it through another day. As the timer in the top right-hand corner approached zero, he tightened his stomach muscles and straightened his back. The mirror snapped its daily photo for his archives and, he suspected, a central database. He clenched his ?st and punched the mirror with all the force he could summon. A thousand pieces ?ew across the room. The holistic guardian of his well-being was dismembered and lying scattered all around him. An enormous sense of relief welled up from deep inside. But. What was that? The fragments of the mirror were continuing their work in isolation and different parts of his body were choosing their own settings. His hands were getting warmer, his feet colder. His heart was racing. His stomach clenched and his calf muscles cramped. And yet as soon as his brain registered a problem, it told him not to worry, immediately overriding all warnings. He screamed as the pain and euphoria of the dissection reached every part of his being. H ABOUT EatingRobots Taken from the new book Eating Robots by Stephen Oram: near-future sciencefiction exploring the collision of utopian dreams and twisted realities as humanity and technology become ever more intertwined. Sometimes funny and often unsettling, these 30 sci-fi shorts will stay with you long after you?ve turned the final page. http://stephenoram.net 96 Good morning, Who do you want to be today? 900 ft Stuff? link in Resources). Python scripts in this tutorial should always be saved in ~/home/.minecraft/mcpipy/, regardless of whether you?re running Minecraft Pi Edition or Linux Minecraft. Be sure to run Minecraft with the ?Forge 1.8? pro?le included in McPiFoMo for your scripts to run correctly. 01 Prerequisites Get yourself a copy of ?Minecraft Stuff?: git clone https://github.com/martinohanlon/ minecraft-stuff Or just visit GitHub and manually download minecraftstuff.py, as that?s all we?ll need for this tutorial. 74 Python prep import import import import import 03 mcpi.minecraft as minecraft mcpi.block as block server time minecraftstuff Connecting Python to Minecraft We?ll want to connect to our Minecraft world: mc = minecraft.Minecraft.create(server.address) Here we?re creating an instance with the variable mc that we can use later on to spawn shapes directly into our open world. We?ll want to use this variable when initiating an instance of ?Minecraft Stuff?: mcdrawing = minecraftstuff.MinecraftDrawing(mc). Now let?s track our current location in-game and we?re good to go: playerPos = mc.player.getTilePos(). 04 Spawning shapes The prefabricated shapes we have available to us are: drawLine, drawSphere, drawCircle and drawFace. We can spawn these shapes in our world with the mcdrawing variable we set up a moment ago, i.e. mcdrawing.drawSphere(), but we?ll need to make sure we pass the coordinates to the function, as well as the type of blocks we?d like to use. For example: mcdrawing.drawSphere(playerPos.x, playerPos.y, playerPos.z, 15, block.DIAMOND_BLOCK.id) 05 Functions and their parameters We have four different shape functions: mcdrawing.drawLine(x1,y1,z1,x2,y2,z2,blockID) mcdrawing.drawSphere(x,y,z,radius,blockID) mcdrawing.drawCircle(x,y,z,radius,blockID) mcdrawing.drawFace(shapePoints,True,blockID) We?ll expand on the drawFace function in a moment. It?s a great tool for ?lling in surfaces. Think of drawFace as a paint bucket from Paint / Photoshop. mcdrawing.drawLine(playerPos.x, playerPos.y + 2, playerPos.z, playerPos.x + 2, playerPos.y + 2, playerPos.z, block. DIAMOND_BLOCK.id) Combined with drawFace, this provides some powerful tools to get creative in our Minecraft worlds, without spending too much time placing blocks. 09 A step in time 10 Status update We imported the time module back in Step 2. That?s so we can slow things down (or speed them up) when needed. When using the drawLine or drawFace functions, it?s best to include a slight pause after each line, otherwise your game could have problems playing catch-up and the lag would result in a messy creation. Just insert a time.sleep(x) after each instance, where x is an integer. A number between 1 and 5 should suf?ce. 06 Custom vectors with drawFace ? part 1 The prefab shapes are great, but you might want to draw a custom shape of your own. We can do that by setting coordinates of each point and then ?lling in the blanks with drawFace. shapePoints = [] shapePoints.append(minecraft.Vec3(x1,y1,z1)) shapePoints.append(minecraft.Vec3(x2,y2,z2)) shapePoints.append(minecraft.Vec3(x3,y3,z3)) 07 Custom vectors with drawFace ? part 2 Now that we?ve set the coordinates of the points, drawFace will connect up the dots and ?ll within the lines with your chosen blockID. mcdrawing.drawFace(shapePoints,True,blockID) For example: mcdrawing.drawFace(shapePoints, True, block.DIAMOND_BLOCK.id). However, if you merely want to draw the outline of the shape, you can change the Boolean from True to False and drawFace will create lines but not ?ll them in. 08 Crossing the line Sometimes you might want to literally just create lines in any given direction, with a speci?c type of block. We can do that with the drawLine function: As your Python script becomes more complex with the addition of different prefab and custom shapes, it?s good practice to comment the code, but also a nice touch to update the current player with a status update. mc.postToChat("Spawning X_Shape in the player world") ?where X_Shape is the name of what you?re spawning. Insert these lines before (or after) creating each shape. Python for Minecraft Pi Using Python, we can hook directly into Minecraft Pi to perform complex calculations, alter the location of our player character and spawn blocks into the game world to create all kinds of creations, both 2D and 3D. We can program pretty much anything from pixel-art, to chat scripts that communicate directly with the player. In this issue we create shapes, both prefabricated and completely custom vector graphics, that spawn in our world at the drop of a hat. With each issue of LU&D we take a deeper look into coding Python for Minecraft Pi, with the aims of both improving our Python programming skills and gaining a better understanding of what goes on underneath the hood of everyone?s favourite voxel-based videogame. www.linuxuser.co.uk 75 Tutorial Google Assistant and ReSpeaker pHAT Raspberry Pi AI Use Google Assistant with your Raspberry Pi to endow it with voice-activated artificial intelligence Nate Drake 02 Return to the Terminal on your Pi and run git clone --depth=1 https://github.com/respeaker/ seeed-voicecard. Switch to this directory with the command cd seeed-voicecard, then launch the installer with sudo ./install.sh 2mic. Once installation is complete, reboot the Pi by running sudo reboot. Reopen a Terminal and run the command aplay -l to is a technology journalist specialising in cybersecurity and Doomsday Devices. list all hardware playback devices. You should see the ReSpeaker listed under card 1 as ?seeed2micvoicec?. Next, run the command arecord -l to list all sound capture devices to check that ?seeed2micvoicec? is listed here too. Resources ReSpeaker Pi drivers http://bit.ly/ GHReSpeaker ReSpeaker 2-Mics pHAT http://bit.ly/ ReSpeakerpHAT ReSpeaker documentation http://bit.ly/ ReSpeakerWiki Tutorial ?les available: ?lesilo.co.uk Google Assistant is a virtual personal assistant, designed to make your life easier by scheduling meetings, running searches and even displaying data. In April this year Google released an SDK (software development kit), allowing developers to build their own Assistant-related hardware. In this guide you?ll learn how to integrate Google Assistant with your Pi and run a demo which can respond to your voice commands. As the Pi doesn?t have a built in microphone, we?ve used the ReSpeaker 2-Mics Pi HAT, which is designed speci?cally for AI and voice applications. The ReSpeaker HAT costs just £12 and is compatible with the Pi Zero as well as the Pi 2B and 3B. However, if you have a USB microphone, you can use this instead. By default, Google Assistant only responds to commands when prompted with a hotword ? ?Hey Google!?. The privacy conscious will be pleased to hear that you can also program the ReSpeaker HAT to listen only when the on-board button is pressed. See the online documentation for tips on how to do this. This tutorial assumes that you have a clean install of the most recent version of Raspbian (Stretch) on your Pi. Once the install is complete, make sure to open Terminal and run sudo apt-get update then sudo apt-get upgrade to bring your system fully up to date. 01 03 Con?gure sound device 04 Adjust sound levels Right-click the volume icon at the top right of your screen and make sure that the voicecard is selected. Right-click once again and choose ?USB Device Settings?. In the new window which opens, click the drop-down menu marked ?Sound Card? and choose the seeed voicecard or another preferred audio device. Next, click ?Make Default? to ensure the Pi will keep using this device next time it restarts. Don?t worry for now about the lack of volume controls; simply click ?OK? to save and exit. Return to the Terminal and run the command sudo alsamixer. This handy utility lets you adjust the volume for playback and recording of all hardware devices. First press F6 and use the arrow to select your chosen audio device, which should be listed under Card 1. Use Return to select it, then press F5 to list all volume settings. Use the left/right arrow keys to select different controls and up/down to adjust individually. Press Esc to quit, then run sudo alsactl store to save the settings. Connect the ReSpeaker Mount the ReSpeaker 2-Mics Pi HAT on your Raspberry Pi, making sure the GPIO pins are properly aligned. If connected properly, the red LED will illuminate. Connect your headphones or speakers to the 3.5mm audio jack on the device. If your speakers aren?t powered, connect another micro-USB cable to the power port on the ReSpeaker to supply enough current. The microphones themselves are located at either end of the HAT, labelled ?Left? and ?Right?. 76 Install device drivers 05 Visit Google Cloud Platform In order to interface your device with Google Assistant, you?ll need to have a Google account. If you don?t have one, head over to https://gmail.com in your web browser of choice and click ?Create an Account? at the top right. Once you?ve done so, visit https://console.cloud.google.com/project and sign in with the account details. Click ?Yes?, then ?Accept?. Next, visit http://bit.ly/GoogleCP. Click ?Enable?. A page will appear entitled ?To view this page, select a project.? Click ?Create? and name your project. Make a note of the project ID, then click ?Create? again. 06 Create product In order to use the Google Assistant API, you need to generate credentials. These are stored in your Pi and let Google know you?re using an authorised device. Click ?Enable? on the API page which has now loaded. Next, visit http://bit.ly/GCP-OAuth. Click ?Con?gure Consent? at the top right and enter a product name such as ?Jane?s Pi Voice Assistant?. If you?re creating a marketable application, ?ll in the homepage, logo and privacy policy ?elds. Click ?Save? at the bottom when you?re done. 07 www.googleapis.com/auth/assistant-sdk-prototype --save -headless. Make sure to replace client_secret_.json with the actual name of the JSON ?le. 09 Enter con?rmation code The Google OAuth tool will now ask you to visit a URL to obtain a con?rmation code. Right-click and copy the link. Open your web browser and right-click the address bar and choose ?Paste and Go?. Click on your own account name and then choose to ?Allow? the voice assistant. The page will generate a code. Copy this from your browser and paste it into the Terminal prompt. If successful, you?ll see a message stating that your credentials have been saved. 10 Start Google Assistant demo In Terminal, run the command sudo apt-get install pulseaudio, then launch it with pulseaudio &. Finally, run the command google-assistant-demo. Make Create client ID Google will return you to the ?Create Client ID? screen. Under ?Application Type?, choose ?Other?, then click ?Create?. You will now see your Client ID and secret key. Ignore these for now and click ?OK?. Google will now return you to the ?Credentials? screen where your client ID will now be listed. Click the down arrow on the right-hand side to save the credentials ?le with the extension .json. Once the ?le is downloaded, open your ?le browser and move it to /home/pi. This will make it easier to access. sure that your headphones or speakers are connected. Use the default hotwords ?Hey Google? to have the Assistant listen, then ask it a standard question such as ?What?s the weather like in London today?? to hear a mini weather forecast. Google Assistant can also play sounds; e.g. ?What does a dog sound like?? It also performs conversions such as asking telling you the value of 100 euros in dollars. Using virtual environments 08 Authorise Google Assistant In your web browser, visit this address: https://myaccount.google.com/activitycontrols. Make sure to switch on ?Web & App Activity?, ?Device Information? and ?Voice & Audio Activity?. Next, open Terminal and run sudo pip install --upgrade google-auth-oauthlib[tool]. You can use this tool to authorise Google Assistant to work with your Pi, using the JSON ?le you downloaded in the previous step. To do this, run the command: google-oauthlib-tool --client-secrets /home/pi/client_secret_.json --scope https:// If you want to use Google Assistant in an embedded device or application, you may wish to create a dedicated virtual environment to store the above demo separately to other projects. Use Terminal to run sudo apt-get install python3.4-venv. Next, choose a name for your virtual environment and create it e.g. python3 -m voice env. If you decide to use a virtual environment, run the above commands after completing Step 7. You can switch to the virtual environment any time with the activate command e.g. source voice/bin/activate. Run deactivate to exit. To run a speci?c program you?ve installed in your virtual environment, enter the command as normal; e.g. google-assistant-demo. www.linuxuser.co.uk 77 Column Pythonista?s Razor Managing data in Python When the amount of data you need to work with goes beyond easy ?at ?les, it's time to move into using a database and a good place to start is SQLite Joey Bernard is a true renaissance man. He splits his time between building furniture, helping researchers with scienti?c computing problems and writing Android apps. Why Python? It?s the of?cial language of the Raspberry Pi. Read the docs at python.org/doc In previous issues, we have looked at how to use data that is stored in ?les using regular ?le I/O. From here, we moved on to looking at how to use pandas to work with more structured data, especially in scienti?c work. But, what do you do when you have data that needs that go beyond these tools, especially in non-scienti?c domains? This is where you likely need to start looking at using a database to manage information in a better way. This month, we will start by looking at some of the lighter options available to work with simple databases. This gives you a connection object that enables interactions with the database stored in the example.db ?le in the current directory. If it doesn?t already exist, the sqlite3 module will create a new database ?le. If you only require a temporary database that needs to live for the duration of your program run, you can give the connection method the special ?lename ?:memory:? to create a database stored solely in RAM. Now that you have a database, what can you do with it? The ?rst step is to create a cursor object for the database to handle SQL statements being applied to the database. You can do so with my_cursor = my_conn.cursor(). The ?rst database thing you will need to do is to create tables to store your data. As an example, the following code creates a small table to store names and phone numbers. "All of the SQLite code is part of your code" In terms of lightweight databases, SQLite is the de facto standard in most environments. It comes as a C library that provides access to a ?lebacked database that is stored on the local disk. One huge advantage is that it does not need to run a server process to manage the database. All of the code is actually part of your code. The query language used is a variant of standard SQL. This means that you can start your project using SQLite, and then be able to move to a larger database system with minimal changes to your code. There is a port to Python available in the module ?sqlite3? which supports all of the functionality. Because it is the standard for really lightweight database functionality, it is included as part of the standard Python library. So you should have it available wherever you have Python installed. The very ?rst step is to create a connection object that starts up the SQLite infrastructure: import sqlite3 78 my_conn = sqlite3. connection('example.db') my_cursor.execute('''CREATE TABLE phone (name text, phone_num text)''') You have to include the data type for each column of the table. SQLite natively supports SQL data types BLOB, TEXT, REAL, INTEGER and NULL. These map to the Python data types byte, str, ?oat, int and None. The execute method runs any single SQL statement that you need to have run against the database. These statements are not committed to the ?le store for the database, however. In order to have the results actually written out, you need to run my_conn.commit(). Note that this method is part of the connection object, not the cursor object. If you have a different thread also using the same SQLite database ?le, it won?t see any changes until a commit is called. This means that you can use the rollback() method to undo any changes, back to the last time commit() was called. This allows you to have you a rudimentary form of transactions, similar to the functionality of larger relational databases. Now that we have a table, we should start populating it with data. The simplest way to do this is to use a direct INSERT statement, e.g.: my_cursor.execute("INSERT INTO phone VALUES ('Joey Bernard', '555-5555')") While this is okay for hard-coded values, you?ll probably have data coming from the user that needs to be entered into the database. In these cases, you should always check this input and sanitise it so there?s no code that can be used for an SQL injection attack. You can do this, then do string manipulation to create the complete SQL statement before calling the execute method. The other option available is to use an SQL statement that contains placeholders that can be replaced with the values stored in variables. This makes the validation of the input data a bit easier to handle. The above example would then look like: my_name = 'Joey Bernard' my_number = '555-5555' my_cursor.execute("INSERT INTO phone VALUES (?,?)", (my_name,my_number)) The values to be used in the SQL statement are provided within a tuple. If you have a larger amount of data that needs to be handled in one go, you can use the executemany() What if SQLite isn?t light enough? function, available in the cursor object. In this case, the SQL statement is structured the same as above. The second parameter is any kind of iterable object that can be used to get a sequence of values. This means that you could write a generator function if your data can be processed that way. It is another tool available to automate your data management issues. Now that we have some data in the database, how can we pull it back out and work with it? The basic SQL statement that is used is the SELECT statement. You can use the following statement to get my phone number. my_cursor.execute("SELECT phone_num FROM phone WHERE name=:who", {"who":'Joey Bernard'}) print(my_cursor.fetchone()) As you can see, you need to call some kind of fetching method in order to get your actual results back. The fetchone() method returns the next returned value from the list of returned values. When you reach the bottom of the list, it will return None. If you want to process returned values in blocks, you can use the cursor method fetchmany(size), where size is how many items to return bundled within a list. When this method runs out of items to return, it sends back an empty list. If you want to get the full collection of all items that matched your SELECT statement, you can use the fetchall() method to get a list of the entire collection. You do need to remember that any of the methods that return multiple values still start wherever the cursor currently is, not from the beginning of the returned collection. Sometimes, you may need to add some processing functionality to the database. In these cases, you can actually create a function that can be used from within other SQL statements. For example, you could create a database function that returns the sine of some value. import math my_conn.create_function("sin", 1, math.sin) cursor2 = my_conn.cursor() cursor2.execute("SELECT sin(?)", (42,)) print(cursor2.fetchone()) There is a special class of database functions, called aggregators, that you may wish to create, too. These take a series of values and apply an aggregation function, like summing, over all of them. You can use the create_aggregate() method to register a class to act as the new aggregator. The class needs to provide a step() method to do the aggregation calculation and a ??λ?????? method that returns the ?nal result. One last item you may want to be able to do is to have larger blocks of SQL statements run against your data. In this case, you will want to use the cursor object?s executescript() method. This method takes a string object that contains an entire script and runs it as a single block. One important difference here is that a commit() call is made just before your script is run. If you need to be able to do rollbacks, this is an important caveat to keep in mind. When you start to have more complicated queries, you may need to track where your results came from. The description property of the cursor object returns the column names for the last executed query. When you are all done, you should always call the close() method of the connection object. But, be aware that a commit is not done automatically. You will need to call it yourself before closing. This ensures that all of your transactions are ?ushed out to disk and the database is in a correct state. Now you can add more robust data management to your code. Sometimes, even SQLite may not be lightweight enough, depending on your speci?c requirements. In these cases, you do have another option. There is a very old ?le-backed database from the earliest days of UNIX, called dbm. Dbm databases store data as a set of key-value pairs within a ?le on the ?le system. To start, you will need to open a database with code like that given below. import dbm db = dbm.open('example.db', 'c') This opens the database in the ?le example.db, or creates it if it doesn?t already exist. You can insert a value to a given key, or get a value based on a key. Below is an example of storing a name/phone number pair. db['Joey Bernard'] = '555-5555' When you do the query, you need to remember that everything is stored as byte strings, so you will need to use those to get values. my_number = db.get(b'Joey Bernard') There are two more advanced variants available, gdbm and ndbm. They each add some further functionality above and beyond that provided by the basic dbm implementation. One important thing to note is that the ?le formats for the different variants are not compatible. So if you create a database with gdbm, you will not be able to read it with ndbm. There is a function, named whichdb(), that will take a ?lename and try to ?gure out which type of dbm ?le it is. Gdbm has methods to easily allow you to traverse the entire database. You start by using the ????????? method to get the ?rst key in the database. You can then travel through the entire database by using the method nextkey(key). Gdbm also provides a method, named reorganize(), which can be used to collapse a database ?le after a number of deletions. Because dbm, and its variants, store data as sets of key/value pairs, it maps quite naturally to the concepts around dictionaries. You can use the same syntax, including the ?in? keyword, from dictionaries when you work with any of the dbm variants. These modules allow you to have data stores that you can use to store simpler data structures within your own programs. www.linuxuser.co.uk 79 ;OLZV\YJLMVY[LJOI\`PUNHK]PJL techradar.com 81 Group test | 86 Hardware | 88 Distro | 90 Free software Antergos Chakra GNU/Linux Manjaro Linux RevengeOS GROUP TEST Arch-based distributions Experience the bene?ts of the venerable Arch Linux in the comforts of a desktop distribution that takes the pain out of that installation Antergos A multilingual distribution that?s proud of the fact that it?s designed with simplicity in mind. Antergos ships with custom artwork and claims it provides a ready-to-use system that doesn?t require any additional steps once it?s been installed. https://antergos.com Chakra GNU/Linux One of the two most popular Arch-based distributions designed for desktop users, Chakra uses a half-rolling release cycle. The distro marks certain packages as core that only receive updates to ?x security issues while other apps are on a rolling-release model. https://chakralinux.org Manjaro Linux RevengeOS The other popular Arch-based desktop distribution, Manjaro, uses its own set of repositories. The distribution is available with three different desktops ? Xfce, KDE and Gnome ? and the latest release has been announced as the project?s last to offer support for 32-bit hardware. https://manjaro.org The oddly-named distribution is all about choice ? starting from its six desktop environments to the default apps. RevengeOS? goal is to provide an easy-toinstall Arch distribution while preserving the power and customisation offered by its Arch Linux base. http://revengeos.weebly.com www.linuxuser.co.uk 81 Review Arch-based distributions Antergos Chakra Linux Makes good use of Arch?s large repos to offer seven desktops A perfect rendition of KDE with an app selection to match Q You can enable the Arch User Repository (AUR), in addition to Antergos? own repo, from the very useful Cnchi installer Q Chakra has several custom repositories, including a communitymaintained repository that?s inspired by the Arch User Repository Installation Installation The home-brewed, multilingual Cnchi installer does a good job of anchoring the distro. It allows you to choose from the seven supported desktops and can also install an Arch-like, CLI-only base. The partitioning step is intuitive enough for new users and also helps advanced ones use ZFS, setup LVM and home on a separate partition. Chakra uses the distribution-independent Calamares installer. It?s intuitive and can be navigated with ease. Like Cnchi, you can ask Calamares to automatically partition the disk or let you modify partitions manually. However, Calamares doesn?t offer as many options as Cnchi installer. Pre-installed apps Pre-installed apps Antergos has the usual slew of productivity apps. By default, the distro uses the Chromium web browser but you can install Firefox during installation where you also get options to pull in LibreOf?ce and Steam. The default Gnome desktop also includes Gnome tweaks and dconf editor to tweak the Gnome desktop. Chakra expressly bills itself as a KDE-centric distribution, which is why it includes a whole gamut of KDE apps including the complete Calligra of?ce suite. It also includes other Qt apps such as the Bomi media player and the lightweight QupZilla browser in place of a more feature-rich browser like Firefox. Usability Usability Besides the project?s Cnchi installer, there are no custom tools of the project. The distro allows you choose between cinnamon, deepin, gnome, KDE, Mate, Openbox and Xfce. All desktops come with their own con?guration tools and they all have been tweaked to look very pleasing with a wide selection of desktop wallpapers, along with the Numix Frost theme. Chakra uses Octopi for package management and for keeping your installation updated. It also includes a cache cleaner and a repository editor. In terms of overall usability, KDE has always had a tendency to overwhelm the new user, which makes Chakra best suited for those who are already familiar and comfortable with the desktop. Help & support Help & support The project has very active multilingual forum boards with separate sections for installation, resolving newbie queries, applications and desktop environments, pacman & package upgrade issues, etc. There are several useful articles in the well categorised wiki and you can also seek advice on the project?s IRC channel. The project has recently overhauled its support infrastructure and the developers now engage with the users via the community section on the website. You can still ?nd the link to the old wiki that has several useful articles to help install and manage the distribution, while others only have skeletal content. Overall Overall Antergos is an aesthetically pleasing distribution. Its Cnchi installer does a wonderful job and helps users customise various aspects of the installation including app selection. 82 8 An Arch-based, 64-bit-only rolling release distro, Chakra?s objective is to give its users the ultimate KDE experience. It?s an excellent desktop distro for those whose demands align with its goals. 7 Manjaro Linux RevengeOS Helps all users leverage the power of Arch with its powerful custom tools A lightweight distribution with tools to help you choose and customise Q With MSM you can install proprietary drivers for connected hardware and even switch between available kernels with a single click QWith the distros? Software Installation Tool you can install popular productivity and multimedia apps with a single click Installation Installation Manjaro is available in three of?cially supported ?avours and while the project doesn?t recommend one over another, we?ve used the Xfce edition that is listed ?rst on the download page. All editions use a customised Calamares installer that enables you encrypt the partition in which you plan to install the distribution. RevengeOS uses its own home-brewed Nemesis installer. It offers three types of installs ? Normal, OEM, and StationX. The installer is fairly intuitive to navigate and allows you choose between six desktop environments. The partitioning step offers an automatic installation option but just ?res up Gparted for manual partitioning. Pre-installed apps Pre-installed apps Manjaro includes all the usual mainstream popular apps such as the LibreOf?ce suite, GIMP, VLC, Firefox, Thunderbird, Steam client, etc. The remaining space is stuffed with handy utilities from the Xfce stable. If you need more apps from the repos, the Xfce edition uses Pamac for package management and handling updates. During installation you get options to install certain useful software such as LibreOf?ce, Wine, Steam client, etc. Besides these, the distro contains the usual collection of apps that ship with the desktop you?ve installed. We opted for its customised OBR Openbox desktop that installs several Xfce tools and some extras like the Gufw ?rewall. Usability Usability Manjaro is very usable since it bundles the regular apps instead of esoteric alternatives. The Xfce desktop follows the conventional desktop metaphor and will suit a large number of potential users. Another plug is the project?s custom tools ? especially the Manjaro Settings Manager ? that allow users really take advantage of the Arch base. The distro boots to a welcome screen that points to scripts to remove the pre-installed virtualbox modules and another to install proprietary Nvidia drivers. There?s also a custom control centre for accessing various custom tools for managing all aspects of the system, from changing wallpapers and docks to installing proprietary codecs and switching kernels. Help & support Help & support Manjaro trumps the lot with a channel on Vimeo with over two dozen videos on various aspects of the distribution and its development. Their wiki is fairly detailed and well categorised, and the project hosts multilingual forums and IRC channels. The distro also bundles a detailed PDF user guide. The welcome screen has links to join the project?s Google+ community page and another to the distro?s fairly active forum boards. Here you?ll ?nd boards for posting problems regarding installation and any issues with the apps. The website also has a wiki with some articles that you may ?nd useful. Overall Overall Manjaro?s developers have gone to great lengths to help users experience the power of Arch. With its custom tools and bundled apps, the ready-to-use desktop scores highly in both form and function. 9 Over?owing with custom tools and scripts, RevengeOS is a well crafted distribution that does a wonderful job of handholding new users and helping them get to grips with Arch. 8 www.linuxuser.co.uk 83 Review Arch-based distributions In brief: compare and contrast our verdicts Antergos Installation Its installer helps you pick a desktop and has smart partitioning options. Preinstalled apps You can use the installer to grab apps like Firefox, LibreOf?ce & Steam. Usability All of the supported desktops have been tweaked to look good and are very usable. Help & Support Offers enough options to help you ?nd your way around any possible issues. Overall A smart-looking distro with an installer that helps customise the install. Chakra GNU/Linux 8 Uses the Calamares installer that?s not as functional as Cnchi but works well. 8 Takes pride in being a KDE-only distro and packs in all sorts of Qt apps. 8 The inclusion of Qtonly apps over some mainstream ones can be an issue for some. 8 It?s new but the recently overhauled support section will ?nd you a solution. 8 The ultimate Archbased distro for users looking for a pure KDE experience. Manjaro Linux 7 The Calamares installer offers users the option of encrypting the installation partition. 8 Includes all of the popular apps you?d expect in a modern desktop distro. 7 The custom tools and the bouquet of apps make it easy to use and tweak. 7 The usual textbased options are complemented by videos on the Vimeo channel. 7 Allows users of all skill levels to experience the power and ?exibility of Arch. RevengeOS 8 Its Nemesis installer is very usable and supports three types of installs. 8 9 The installer helps you to pick the desktop as well as popular, everyday apps. 8 9 You can mould the installation as per your needs with its custom control centre. 9 9 You get a Google+ page and a couple of boards on the forum to post your queries. 7 9 Essentially a one-man show that includes a great many tools to shape the installation. 8 AND THE WINNER IS? Manjaro Linux After Ubuntu, Arch Linux is one of the most popular projects that?s used as a base for creating all kinds of niche as well as general purpose distributions. In this group test, we?ve included the ones that focus on helping new and inexperienced users uncover the power of Arch. These distros help save them the time and effort investment that?s typically required in deploying an Arch install. Chakra ruled itself out of contention for its affection towards KDE users, which, with its myriad of options, isn?t always the best environment for inexperienced users. RevengeOS and its gamut of custom tools make for a pleasant experience but the distro is a one-man show without a ?xed release schedule. Plus if you don?t like the default theme, tweaking the visuals will take some time and effort that?s best spent elsewhere. In the end, the real contest ? and a close one at that ? was between Antergos and Manjaro. A big plus for Antergos is that it offers the option of multiple desktops during installation. It?s also visually appealing but apart from being a Cnchi installer, there are no other custom tools. 84 Q Use the CLI Manjaro architect tool to install a customised build with your choice of kernel, drivers etc Manjaro, on the other hand, includes the very useful Manjaro Settings Manager that helps users take advantage of the Arch ecosystem without getting into the nittygritties. It?s also chock-full of apps and can be used straight out of the box. The project has its own dedicated software repositories that deliver thoroughly tested and stable software that works well in conjunction with Arch and AUR repositories and can be used for bleeding-edge software. Finally, Manjaro remains one of the very few independently developed projects that is still on the market for both 32-bit and 64bit architectures. Mayank Sharma Not your average technology website EXPLORE NEW WORLDS OF TECHNOLOGY GADGETS, SCIENCE, DESIGN AND MORE Fascinating reports from the bleeding edge of tech Innovations, culture and geek culture explored Join the UK?s leading online tech community www.gizmodo.co.uk twitter.com/GizmodoUK facebook.com/GizmodoUK iStorage DiskAshur Pro 2 Review HARDWARE iStorage diskAshur Pro 2 Price From £195 for 500GB (reviewed £489, 5TB) Website https://istorage-uk.com Specs Capacity: 500GB-5TB Data transfer speed: Up to 148MBps (Read),140MBps (Write) Power supply: USB bus-powered Interface: USB 3.1?up to 5Gbps Hardware data encryption: RealTime Military Grade AES-XTS 256-bit Full-Disk Hardware Encryption Warranty: 2 years Dimensions (W, D, H): 124mmx84mmx20 mm (500GB,1/2TB), 124mmx84mmx28mm (3/4/5TB) Weight: 500GB/1/2TB 225g, 3/4/5TB 331g 86 An expensive portable hard drive on the face of it, but good security has never been cheap For those who carry sensitive information around with them on a daily basis, there?s an everpresent concern with losing the device carrying that precious data (or worse, having it stolen). This is concern addressed by the new iStorage diskAshur Pro2 external HD, a compact storage device designed to work with secure data without the need to install software on all the systems it will meet. The Pro2 retails at £489 ($651), comes in 500GB, 1TB, 2TB, 3TB, 4TB and 5TB capacities, and it was the latter which the ?rm supplied to us for review. The Pro2 is expensive as 5TB external drives go, being quadruple what Seagate asks for its Backup Plus 5TB drive, and more than double LaCie?s rugged Thunderbolt 5TB models. But after removing the drive from the packaging, we realised why iStorage asks so much for it, because this is, without doubt, one of the most glorious pieces of product engineering we?ve had the pleasure to handle. The upper and lower surfaces are cool-to-the-touch metal, and the waistband is softtextured rubber. The drive is just 84mm wide, 124mm long and 20mm deep, dictating that this uses 2.5-inch drives internally to provide 5TB of capacity. As if to underline how much of the cost goes into the engineering and not the capacity, the 2TB model is £329 ($437 approx), only £160 ($212 approx) cheaper. The unit has numerous hardware safeguards to defend against external tampering, bypass attacks and fault injections From a design perspective, two features make this drive special: the ?rst is the built-in USB cable. The cable is only 12cm long when unclipped, but that?s enough to attach it to a laptop or desktop PC. The other standout feature is the built-in numeric pad. This is an integral feature of the security mechanism iStorage has implemented. In addition to the numbers, there are a few special keys for operating the unit when it?s attached to a computer. Above the numeric pad are three LEDs that con?rm the locked condition, and also show drive activity. At the heart of this design is a secure microprocessor (Common Criteria EAL4+ ready) that handles the encryption of the device. This, in theory, means that if the bare drive is extracted from its case, an attacker is no closer to getting to the data. A data thief will need the 7-15 digit numeric password created when the Pro2 was last con?gured. If you fail to enter the correct code enough times then this will result in the drive deleting the encryption key, rendering the contents beyond reach forever, unless you know people in the security services who can crack AES-XTS 256-bit. In addition, the unit also has numerous hardware safeguards to defend against external tampering, bypass attacks and fault injections. Should it detect any attempt to get into the case or tinker with USB, it will trigger a deadlock frozen state, at which point further assault is pointless. Devices with a numeric pad like this usually come with a PC application that you?ll need to install, but the Pro2 is fully self-contained. That allows it to work as well with Linux as many other OSes it supports, such as Android, Chrome, MacOS and Window. You can format it to whatever ?lesystem you use?even one you?ve created yourself. The unit comes with a default admin PIN number de?ned, and you can change that directly using the pad. But the most Bond-esque PIN code you can de?ne is the one that initiates a ?self-destruct? sequence, which performs an internal crypto-wipe where all the PINs and data are erased, and the drive must be reformatted before it can be used again. For the majority of people, the Pro2 has enough in the way of protection. With 145.5MBps reads and 144.8MBps writes, the spinning disk inside the Pro2 has some intent about it. While an SSD would be quicker (and iStorage provides models with those inside, too), those performance levels are good enough, and about as rapid as a PC with hard disk-based storage is likely to be. The unit is also IP56 certi?ed, making it water and dust-resistant, though not waterproof. An extra touch in terms of the physical protection is that the keys on the pad are coated in epoxy. The coating makes it harder to work out what keys are being used on a regular basis. Mark Pickavance Pros Beautiful construction and offers genuine data security. Being self-contained, it will work with any Linux distro Cons You get what you pay for, so it?s expensive and also potentially a little daunting to set up Summary A genuinely secure storage device that?s built to handle physical abuse and nefarious decryption ? but it comes at a premium price. The overall combination of a well-considered security model which is also a superbly engineered device is an alluring one. 9 www.linuxuser.co.uk 87 Review Pop!_OS 17.10 Above Pop is available only for 64-bit architecture with separate releases for Intel/AMD and Nvidia hardware DISTRO Pop!_OS 17.10 Caught out by the dramatic changes in Ubuntu, System76 decides to take matters into its own hands RAM 2GB Storage 20GB Specs CPU: 64-bit Intel or AMD processor Graphics: AMD or Nvidia hardware Available from: https:// system76.com/pop 88 System76 is one of the few retailers to sell computers with 100 per cent Linux compatible hardware pre-loaded with Ubuntu. The desktop may not be Canonical?s core business, but it is at System76?the end of the Unity project affected 91 per cent of its business. In response, it put together a distribution of its own to offer customers a user experience in line with the company?s hardware. The company went distro hopping and settled on the latest Ubuntu with GNOME Shell as the base for its custom distro christened Pop!_OS. Instead of focusing its efforts on mainstream desktop users, System76 is designing a distro geared towards creators, developers and makers: ?If you?re in software engineering, scienti?c computing, robotics, AI or IOT, we?re building Pop!_OS for you,? says CEO, Carl Richell in his blog post announcing the beta release. ?We?ll build tools to ease managing your dev environment. We?ll make sure CUDA is easily available and working.? System76 raised our expectations with its ?Imagine? marketing (https://system76.com/pop) that promises ?the most productive and gorgeous platform for developing your next creation?, even if it adds ?we?re just getting started?. Speaking to Richell on the ?rst full release, he explained that the company?s intention is to establish the ?guiding principles of the distribution, develop the aesthetic, build infrastructure, testing and quality procedures, start a community, documentation, web pages, production (shipping on laptops and desktops at scale) and of course, the release.? Without this background, the ?rst release seems bland and minimalistic given the objectives on its website. ?In four months we?ve laid the infrastructure and direction of our OS. The next six months are Above Sure, Pop is pitched at users who might not be aware of Gnome, but not mentioning the desktop even once might alienate those in the know As a good open source supporter, System76 is also improving upstream code executing on our vision?, explains Carl. We feel it?s important to bear this in mind when assessing the distro, but we can?t review what isn?t there yet. On the software side, the distro is pretty standard desktop fare, with LibreOf?ce, Firefox and Gnome?s default bouquet of applications for viewing images and videos. Compared to Ubuntu, Pop uses Geary for email instead of Thunderbird and doesn?t include the default Ubuntugames, the pesky Amazon integration, Cheese, Transmission, Rhythmbox, and Shotwell. Carl says that in the company?s experience these missing apps aren?t widely used by customers. Pop?s installation is based on Ubuntu?s Ubiquity installer. The only real difference apart from cosmetic ones is that there?s no initial user creation step. This has been moved to a post-install ?rstboot wizard. Pop?s GNOME desktop has been tweaked to deliver a user experience that best suits its customers. GNOME extensions are enabled that display workspaces whenever you bring up the Activities Overview, and another that adds a Suspend button in the Power menu. To protect your privacy, Pop also doesn?t display noti?cations on the lock screen. The System76 UI team has spent considerable time getting the visual details right. The ?at Pop GTK theme based on the Materia GTK+ theme with matching cursor and icons are pleasing. As a good open source supporter, System76 is also improving upstream code. Carl tells us that it?s improved GNOME?s half-tiling window focusing and contributed patches to ensure HiDPI works properly. The one GNOME tweak that made things dif?cult for us were the remapped keyboard shortcuts. Carl tells us that the shortcuts have been worked out with input from software developers, but the tweaks necessitated a visit to the keyboard settings section to reassign them to the familiar GNOME values. Another highlight is the Pop!_Shop app store, which is based on code from elementary OS?s AppCenter. While the distro uses the same repositories as Ubuntu 17.10, System76 handpicks the listed apps. This is why you can?t install apps such as Thunderbird, Evolution, Chromium from the Pop!_Shop. However, you can fetch them using APT. Another application that Pop has borrowed from the elementary OS project is Eddy, the DEB package installer. Carl tells us System76 are considering creating ?application suites? that will help users fetch multiple apps relevant to a particular task. Mayank Sharma Pros An aesthetically pleasing GNOME rendition with useful extensions enabled by default and an intuitive app store. Cons A minimalist distro with a default application selection that so far fails to meet its grandiose objectives. Summary A good-looking distro that doesn?t yet match what it promises on the of?cial website. But much of the work in this ?rst release is in the background and lays the groundwork for future developments that?ll help push the distro as the perfect platform for developers. 7 www.linuxuser.co.uk 89 Review Fresh free & open source software CODE EDITOR CudaText 1.23.0 A feature-rich editor for writing code Linux has no dearth of advanced text editors that moonlight as lightweight IDEs as well and we ran a group test of some of the best a few issues ago (LU&D182, Pg 81). CudaText is one such crossplatform editor that?s primarily designed to write code, but can double up as an advanced text editor. The editor has all the usual coding conveniences, like syntax highlight for several programming languages, including C, C++, Java, JavaScript, HTML, CSS, Python, and more. You also get code completion for some languages like HTML and CSS, code folding, the ability to search and replace with regex, as well multi-caret editing and multi-selection. You can extend the functionality by installing additional plugins that are written in Python. The project releases precompiled binaries for Debian-based distros. On others you?ll have to compile it manually by following the instructions on the wiki. It has a single document that explains all aspects of the app, and is a must-read to get to grips with all the functionalities of the app. If you?ve worked with advanced text/code editors before, you?ll have no issues navigating CudaText?s interface. Above The app has colour themes for the interface. Each has a matching scheme for coding syntax Pros Includes all the common coding conveniences and can be extended with plugins. Cons You?ll have to edit its con?guration ?le to hook it up with the Python interpreter. Great for? Editing code without an IDE. http://uvviewsoft.com/ cudatext/ DIVE LOG TOOL Subsurface 4.7.1 Log and analyse all your scuba dives with ease Linus Torvalds likes to track whatever he does. He wrote Git to keep track of the kernel development and Subsurface to log his dives. Torvalds likes to don the wetsuit and plunge underwater whenever he isn?t busy hacking away at the Linux kernel. He couldn?t ?nd a good app to log his dives, so naturally he wrote one himself. Simply put, Subsurface helps keep track of scuba dives. You can use it to import data from one of the supported dive computers, usually via USB. Once your data is imported, you can view and edit dive details from the intuitive user interface. The app also enables you to log dives manually. The app shows a dive pro?le graph that plots dives as a line graph. 90 It shows ascent and descent speeds along with the depths. Certain events, such as the begin of a decompression, are also marked. The app also records details about the diving equipment, and can calculate various stats about multiple dives. There?s a detailed user manual to help acquaint users, as well as a video tutorial. The latest version sports some user interface changes, like a new map widget. Support for importing dive data from Shearwater desktop, DL7, Datatrak and in other third-party formats has also been improved, and the version has experimental support for Bluetooth LE dive computers. The project lists binaries on its Downloads page for several popular distros, and there?s an AppImage that?ll work on any Linux distro. Pros Very easy to install and works with a large number of dive computers. Cons You?ll have to read through its documentation to discover all its features. Great for? Analysis of dive data. https://subsurfacedivelog.org DESKTOP ENVIRONMENT LXQt 0. 1 2.0 Bring old workhorses to life with this lightweight desktop environment The Lightweight Qt Desktop Environment?called LXQt for short?will draw a graphical user interface without consuming too many resources. The desktop environment is a mix of GTK-based lightweight desktop, LXDE, and RazorQt, which was an equally lightweight?but less mature?desktop that used the Qt toolkit. The recent releases of the mainstream desktop environments such as Gnome and KDE have put them out of the reach of low-spec machines, which is why many popular distros have a LXQt-based ?avour in their arsenal of releases. LXQt is also popular with users of newer more powerful machines, as it helps free the resources for more CPU-intensive tasks such as video processing. Pros The latest version of the desktop includes better support for HiDPI displays. The new release ships with a new Open/Save File dialog, and includes support for icon themes that use the FollowsColorScheme KDE extension to the XDG icon themes standard. Behind the scenes, the developers have also improved the shutdown/reboot process by shutting down all LXQt components before allowing systemd to do its job. There have been some important architectural changes too. The lxqt-common package has been dropped, and its components split into several other packages, such as the newly introduced LXQt-themes package. The project?s website has a list of distros that ship the LXQt desktop either in a spin or via their repositories. A fully functional and smartlooking desktop that doesn?t consume too many resources. Cons Doesn?t include the bells and whistles you get with the mainstream desktops environments. Great for? Busy and old PCs. https://github.com/pixlra/ playuver IMAGE CONVERTER Converseen 0.9.6.2 Automate repetitive image processing tasks If you?ve worked with images, either professional or sorting through vacation clicks, you know a lot of image processing work is monotonous?repeatedly converting, resizing and rotating images to make them suitable for print or passing around. Although you can use virtually any image viewing or editing app, from DigiKam to GIMP, for this task, you?ll save yourself hassle by employing a dedicated batch conversion tool like Converseen. Converseen is a straightforward frontend to the command-line conversion utilities in ImageMagick. You can use the app to convert images to and from over 100 formats, rotate and ?ip them, change their dimensions and rename them in a fraction of the time it would take to perform these tasks manually. You?ll ?nd installable binaries for Ubuntu, Fedora and OpenSUSE on the project?s website, along with simple installation instructions. The work?ow is pretty straightforward?click the Add Images button to select any number of images you want to convert. Then scroll down the Action panel on the left to specify how you want to manipulate the image, and hit the Convert button to begin the process. Above Converseen exposes only a fraction of the image manipulation tricks you can do with ImageMagick Pros Straightforward interface that helps automate mundane image editing tasks Cons Great for? Exposes a very limited subset of the power of the CLI tools it?s based on. Resizing, rescaling and rotating. http://converseen.fasterland. net www.linuxuser.co.uk 91 Web Hosting Get your listing in our directory To advertise here, contact Chris chris.mitchell@futurenet.com | +44 01225 68 7832 (ext. 7832) RECOMMENDED Hosting listings Featured host: Use our intuitive Control Panel to manage your domain name www.thenames.co.uk 0370 321 2027 About us Part of a hosting brand started in 1999, we?re well established, UK based, independent and our mission is simple ? ensure your web presence ?just works?. We offer great-value domain names, cPanel web hosting, SSL certi?cates, business email, WordPress hosting, cloud and VPS. What we offer � Free email accounts with fraud, spam and virus protection. � Free DNS management. �
1/--страниц