tag:blogger.com,1999:blog-80087786836441548642024-03-08T00:07:15.548-05:00Unix UniqueA collection of little fixes.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-8008778683644154864.post-11971300612530857912018-07-09T16:58:00.004-04:002018-07-09T17:00:36.912-04:00Linux Network Emulator Custom Delay Distribution<p>The Linux <strong>netem</strong> tool enables simulation of various network conditions including packet loss, latency, and jitter. It is a useful tool for testing new transport protocols, such as QUIC or TCP with BBR congestion control, under adverse network conditions. A brief tutorial on <strong>netem</strong> can be found here: <a href="https://wiki.linuxfoundation.org/networking/netem">https://wiki.linuxfoundation.org/networking/netem</a></p>
<h2 id="netem-basics">netem Basics</h2>
<p>Setting up <strong>netem</strong> is simple:</p>
<pre><code>tc qdisc add dev eth0 root netem
</code></pre>
<p>One can simulate a network with 1% packet loss and a latency of 100ms as follows:</p>
<pre><code>tc qdisc change dev eth0 root netem delay 100ms loss 1%
</code></pre>
<p><em>Important note: <strong>netem</strong> policies are applied on outbound traffic only. With the command above, only outgoing IP packets will be delayed by 100ms and dropped with 1% probability. A simple ping test can verify your settings.</em></p>
<h2 id="delay-distribution">Delay Distribution</h2>
<p>The <strong>netem</strong> tool can also be used to introduce non-uniform delay latency. When a single argument is passed in for delay, the delay is uniform. In our example above, every packet is delayed by exactly 100ms. In the real world, and especially on wireless networks like WiFi or Mobile 3G and 4G LTE, delay is not uniform. With <strong>netem</strong> we can also emulate a delay distribution.</p>
<pre><code>tc qdisc change dev eth0 root netem delay 100ms 10ms loss 1%
</code></pre>
<p>By adding a second argument to the delay, we have specified a jitter. This means that the average or mean latency will be 100ms but packets will vary +/- 10ms. More precisely, packets will experience a mean latency of 100ms with a standard deviation of 10ms with a <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal distribution</a>.</p>
<p>Other probably distributions are available and can be used by specifying the <code>distribution</code> argument:</p>
<pre><code>tc qdisc change dev eth0 root netem loss %1 delay 100ms 10ms distribution pareto
</code></pre>
<p>In the above command, a <a href="https://en.wikipedia.org/wiki/Pareto_distribution">pareto distribution</a> with a standard deviation of 10ms is used instead of the default normal distribution.</p>
<p>The full list of distribution tables can be found under <code>/usr/lib/tc/</code> or <code>/usr/lib64/tc</code>:</p>
<pre class=" language-bash"><code class="prism language-bash">$ <span class="token function">ls</span> /usr/lib64/tc/*.dist
/usr/lib64/tc/experimental.dist /usr/lib64/tc/normal.dist /usr/lib64/tc/pareto.dist /usr/lib64/tc/paretonormal.dist
</code></pre>
<p>However, even these built-in distribution tables may not simulate real world conditions. For example, consider the following the following histogram of ping packets sent between two stops on subway ride:<br>
<img src="https://i.imgur.com/EHK44Jb.png" alt="Subway Ping RTT"><br>
The average RTT is 76ms and the standard deviation is 120ms. A normal distribution would not accurately represent this histogram very well.</p>
<p>Fortunately, <strong>netem</strong> allows for custom delay distributions. In the documentation they mention this explicitly:</p>
<blockquote>
<p>The actual tables (normal, pareto, paretonormal) are generated as part of the <a href="https://wiki.linuxfoundation.org/networking/iproute2" title="networking:iproute2">iproute2</a> compilation and placed in /usr/lib/tc; so it is possible with some effort to make your own distribution based on experimental data.</p>
</blockquote>
<p>The remainder of this post covers the details of what “some effort” implies.</p>
<h2 id="distribution-table-file-format">Distribution Table File Format</h2>
<p>The built-in distribution tables offer a clue as to the proper structure of a custom distribution. By examining the <code>normal.dist</code> file we can determine the right format for a custom distribution file.</p>
<p>Here are the first few lines of the file. It starts the value -32768 (min value for a a 16-bit signed int). The values are arranged into eight columns for readability. The values span from -32768 to 32767. In all there are 4096 values. You can think of each value as a random sample taken from the distribution.</p>
<pre class=" language-bash"><code class="prism language-bash"><span class="token comment"># This is the distribution table for the normal distribution.</span>
-32768 -28307 -26871 -25967 -25298 -24765 -24320 -23937
-23600 -23298 -23025 -22776 -22546 -22333 -22133 -21946
-21770 -21604 -21445 -21295 -21151 -21013 -20882 -20755
-20633 -20516 -20403 -20293 -20187 -20084 -19984 -19887
-19793 -19702 -19612 -19526 -19441 -19358 -19277 -19198
-19121 -19045 -18971 -18899 -18828 -18758 -18690 -18623
-18557 -18492 -18429 -18366 -18305 -18245 -18185 -18127
-18070 -18013 -17957 -17902 -17848 -17794 -17741 -17690
-17638 -17588 -17538 -17489 -17440 -17392 -17345 -17298
-17252 -17206 -17160 -17116 -17071 -17028 -16984 -16942
-16899 -16857 -16816 -16775 -16735 -16694 -16654 -16615
</code></pre>
<p>A histogram of the values in this file would reveal a normal distribution with a mean of zero and a standard deviation of 8191.</p>
<h2 id="custom-distribution-file">Custom Distribution File</h2>
<p>Creating a custom distribution involves generating a <code>.dist</code> file and placing it in the <code>/usr/lib/tc</code> (or <code>/usr/lib64/tc</code>) directory. We need to generate 4096 samples that are representative of our distribution within the -32768 to 32767 range. As we design our distribution, the mean should be zero and the standard deviation should be 8191. When the distribution is used by<strong>netem</strong>, the first argument of the delay, the average delay, will be used as the average and the the second argument will be used as the standard deviation. In other words, <strong>netem</strong> will scale our distribution to match the two parameters we provide.</p>
<h3 id="simple-uniform-distribution">Simple Uniform Distribution</h3>
<p>Suppose that we want to create a distribution that spreads out delay values uniformly between 90ms and 110ms. We could create a simple script that prints -32768 (the starting value) and then 4095 values evenly spaced between -8192 and 8191.</p>
<pre class=" language-python"><code class="prism language-python"><span class="token comment">#!/bin/env python</span>
<span class="token keyword">print</span> <span class="token operator">-</span><span class="token number">32768</span>
<span class="token keyword">for</span> x <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token operator">-</span><span class="token number">8192</span><span class="token punctuation">,</span> <span class="token number">8188</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span> x
</code></pre>
<p>The output of our python script can then be formatted using a little awk:</p>
<pre class=" language-bash"><code class="prism language-bash">python uniform.py <span class="token operator">|</span> <span class="token function">awk</span> <span class="token string">'ORS=NR%8?FS:RS'</span> <span class="token operator">></span> uniform.dist
</code></pre>
<pre class=" language-bash"><code class="prism language-bash">$ <span class="token function">head</span> uniform.dist
-32768 -8192 -8188 -8184 -8180 -8176 -8172 -8168
-8164 -8160 -8156 -8152 -8148 -8144 -8140 -8136
-8132 -8128 -8124 -8120 -8116 -8112 -8108 -8104
-8100 -8096 -8092 -8088 -8084 -8080 -8076 -8072
-8068 -8064 -8060 -8056 -8052 -8048 -8044 -8040
-8036 -8032 -8028 -8024 -8020 -8016 -8012 -8008
-8004 -8000 -7996 -7992 -7988 -7984 -7980 -7976
-7972 -7968 -7964 -7960 -7956 -7952 -7948 -7944
-7940 -7936 -7932 -7928 -7924 -7920 -7916 -7912
-7908 -7904 -7900 -7896 -7892 -7888 -7884 -7880
</code></pre>
<p>Copy <code>uniform.dist</code> to <code>/usr/lib/tc/</code> (or <code>/usr/lib64/tc</code>) to “install” the distribution.</p>
<pre class=" language-bash"><code class="prism language-bash">$ <span class="token function">cp</span> uniform.dst /usr/lib64/tc/
</code></pre>
<p>Once copied, you can now run <code>tc</code> commands using <code>uniform</code> as a distribution name. For example, to generate a uniformly distributed random delay between 90ms and 100ms run the following:</p>
<pre><code>tc qdisc change dev eth0 root netem delay 100ms 10ms distribution uniform
</code></pre>
<h3 id="generating-distributions-from-real-world-samples">Generating Distributions from Real World Samples</h3>
<p>Creating a more complex distribution table requires a more complex python script. Real world data can be used to generate a distribution by collecting 4095 samples from a real world environment. Calculate the average and the standard deviation over the samples. Next subtract the average from all samples. Finally divide each sample by the standard deviation and multiply by 8191. Remove any samples that are outside the -32768 to 32767 range.</p>
<h3 id="generating-distributions-from-histograms">Generating Distributions from Histograms</h3>
<p>Alternatively, it’s possible to generate samples based on a histogram. Consider the following script which defines a few “buckets” and assigns a certain number of samples to each bucket.</p>
<pre class=" language-python"><code class="prism language-python">buckets <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">(</span><span class="token number">209</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">6144</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">4096</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">419</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">4096</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2048</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">1419</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2048</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">1319</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">2048</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">419</span><span class="token punctuation">,</span> <span class="token number">2048</span><span class="token punctuation">,</span> <span class="token number">4096</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">205</span><span class="token punctuation">,</span> <span class="token number">2048</span><span class="token punctuation">,</span> <span class="token number">4096</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token number">105</span><span class="token punctuation">,</span> <span class="token number">24576</span><span class="token punctuation">,</span> <span class="token number">32768</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token keyword">print</span> <span class="token operator">-</span><span class="token number">32768</span>
<span class="token keyword">for</span> bucket <span class="token keyword">in</span> buckets<span class="token punctuation">:</span>
start <span class="token operator">=</span> bucket<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>
end <span class="token operator">=</span> bucket<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span>
num <span class="token operator">=</span> bucket<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
step <span class="token operator">=</span> <span class="token punctuation">(</span>end <span class="token operator">-</span> start<span class="token punctuation">)</span> <span class="token operator">/</span> num
<span class="token keyword">for</span> x <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>bucket<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span> start <span class="token operator">+</span> x<span class="token operator">*</span>step
</code></pre>
<p>Each bucket is defined with the number of samples it should include as well as a start and end range.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Linux <strong>netem</strong> is a useful tool for simulating network conditions. However, network delay, latency, or round-trip time rarely follow one of the built-in distribution types (normal, pareto, or paretonormal). Generating new distributions that match either real-world or artificial conditions is possible with some simple scripting work described above.</p>
Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-59129329271037610252015-03-18T13:22:00.005-04:002015-03-18T13:22:43.235-04:00RHEL 7: Disable FirewallUnlike previous versions, RHEL 7 uses the firewalld service instead of the iptables service for its firewall. The firewall can be completely disabled with the following systemctl commands:<br/>
<br/>
<code>
systemctl disable firewalld<br/>
systemctl stop firewalld
</code><br/>
<br/>
Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-1824808335554484212015-03-18T13:22:00.001-04:002015-03-18T13:22:10.275-04:00CentOS 7: Disable FirewallUnlike previous versions, CentOS 7 uses the firewalld service instead of the iptables service for its firewall. The firewall can be completely disabled with the following systemctl commands:<br/>
<br/>
<code>
systemctl disable firewalld<br/>
systemctl stop firewalld
</code><br/>
<br/>
Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-86192105506614166612015-01-30T16:16:00.000-05:002015-01-30T16:16:40.506-05:00Gerrit git review fails for initial empty repositoryGerrit is a code review tool built on and for git - a distributed version control system. When creating a project with gerrit, the user has an option to either create a completely empty repository (where an existing project could be imported) or to create a repository with an initial empty commit, as follows:<br />
<br />
<code># gerrit create-project --empty-commit --name myproject</code><br />
Without the initial empty commit, the underlying git repository is left empty without any revisions or branches.<br />
<br />
Another tool used in conjunction with gerrit is git-review. This tool simplifies creation and management of reviews from the command line.<br />
<br />
If a user attempts to run git-review on a repository while using the completely empty gerrit repository as the remote, the command may fail since there is no master branch yet created in the remote repository.<br />
<br/>
<code><pre># git review
Errors running git rebase -i remotes/origin/master
fatal: Needed a single revision
invalid upstream remotes/origin/master</pre></code><br/>
<code><pre># git review -R
Had trouble running git log --decorate --oneline HEAD --not remotes/origin/master --
fatal: bad revision 'remotes/origin/master'</pre></code></br/>
For the first commit, the user can circumvent git-review and instead push directly to gerrit using git.<br/>
<br/>
<code># git push origin HEAD:refs/for/master</code><br/>
As a result, a new review will be created in gerrit which can then be merged into the project's master branch. After an initial commit is merged into the master branch, the git-review command can be used as usual. <br/>
<br/>
If the above command fails, complaining about the commit message "missing Change-Id in commit message footer" then simply amend the suggested Change-Id to the the commit message.<br/>
<br/>
<code># git commit --amend</code><br/>Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-42161590982357104462014-09-12T02:07:00.000-04:002014-09-12T02:07:30.139-04:00What is OpenStack?OpenStack is a collection of open source projects designed for creating and managing cloud infrastructure.<br />
<br />
Imagine owning a data center. You have a large open space with plenty of power and cooling capacity. You have negotiated a deal with your Internet Service Provider for a few high bandwidth connections to the Internet. You have gone through the trouble of filling that data center with rows upon rows of general purpose servers and networking equipment to interconnect them. Now what?<br />
<br />
The data center could become your personal super computer - graphics rendering, calculating pi, mining bitcoin, etc... or, you could rent it out and bring in some revenue, but who would rent the whole data center with all those servers all at once? Instead of wooing a single tenant for your space, why not rent out the data center in sections to multiple tenants? <br />
<br />
Renting out physical infrastructure is how some hosting companies - like Rackspace - got their start. Tenants could work closely with operators to configure their section of the data center according to the tenant's needs. In return, the tenant would pay for the infrastructure and support. <br />
<br />
For our data center, renting physical infrastructure is a good starting business model. However, we can do better. First, consider all the manual labor involved with hand holding each tenant and helping to configure the infrastructure according to their needs. Can't it be automated? Second, think of all the wasted capacity - a tenant may only need their server for 30 hours a month to run a nightly report even though the tenant is paying for a full month of 24/7 access. Outside of those 30 hours, your server (your investment) is dormant. If we know our tenant is not using the full capacity of a server, why not double book the server and charge another tenant for access to the same resource?<br />
<br />
Consider a new model with the following characteristics:<br />
<ol>
<li><i>On-demand Self-service</i> - no more hand holding tenants through the configuration process. The whole thing is automated and managed via a web interface, available 24/7.</li>
<li><i>Resource Pooling</i> - multiple tenants can rent the same infrastructure with the help of virtualization. Virtual Machines (VMs) give the illusion of a single machine (server) for each tenant. In reality, multiple VMs share a single physical machine and a special operating system (called a hypervisor) quickly switches between VMs to maintain the illusion.</li>
<li><i>Rapid Elasticity - </i>allow tenants to add or remove servers on a whim, as needed. It's all automated and virtualized anyway, so why not?</li>
<li><i>Broad Network Access - </i>something we were already providing in our rental model, but important enough to carry over to the new model. </li>
<li><i>Measured Service </i>- with the above characteristics, accurately billing tenants becomes a nightmare without automated metering of resource consumption. Instead of charging per server per month, we'll switch to more granular pricing: per hour for CPU consumption, per GB for storage, and per Gb for network bandwidth.</li>
</ol>
<div>
The new model is attractive for all kinds of businesses and use cases, big or small. Some tenants will swoop in, rent 1000 machines for an hour for heavy number crunching on-demand, and then leave. Others will setup low traffic web sites with low levels of sporadic resource consumption. A few will have big operations with spikes in usage where their applications spread from 10s to 100s of servers and then fall back again - elastically. </div>
<div>
<br /></div>
<div>
According to NIST, this service is <a href="http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf">by definition</a> a "cloud".<br />
<br />
The business (or service) model is "Infrastructure as a Service" (IaaS). Infrastructure refers to the (virtual) servers and networking equipment which the data center operator is offering to tenants. It is a service since the tenants are not purchasing this equipment - they are renting.<br />
<br />
The deployment model is a "public cloud" - anyone in the world can access our website and rent out virtual machines - it is publicly available. In another scenario, imagine you are the CTO of a Fortune 500 company and are responsible for the many data centers owned by that company. It may still be beneficial to use the Cloud Computing model to manage your data center resources among the many groups within the company. However, these resources are not available to the general public or even to other companies. In such a scenario, the deployment model is a "private cloud".<br />
<br />
What about OpenStack?<br />
<br />
As a highly intelligent data center owner and operator, you may be savvy enough to write your own software to expose a web interface to your tenants which allows them to create user accounts, upload virtual machine images, spawn virtual machines, setup network connectivity for each new machine, handle security between tenants, provide resource usage reporting, and so on. If not, or if you are too busy to undertake such a task, then OpenStack is the answer.<br />
<br />
<a href="http://openstack.org/">OpenStack </a>is a collection of open source projects designed for creating and managing cloud infrastructure. It includes projects for handling the virtual machine life cycle (nova), storing and managing virtual machine images (glance), managing tenant, user, and admin accounts (keystone), exposing a web interface for end users (horizon), handling virtual networking (neutron), handling block storage devices (cinder), and several more.<br />
<br />
As a data center operator, you may install OpenStack software on each server in your data center. Some servers should be designated for "compute" or "storage" - meaning that these servers will be available for tenants to start their virtual machines and create virtual storage devices. Other servers may be designated for networking, for running tenant network services like DHCP and DNS. Finally, a few servers should be earmarked for the Cloud Controller. The Cloud Controller receives requests for creating and destroying virtual machines and other virtual resources. It delegates the real work to one of the many compute, storage, or network nodes in the data center.<br />
<br />
The OpenStack projects are all open source and rely heavily on other existing open source projects. The majority of the OpenStack code base is written in the Python programming language. Most OpenStack projects exposes a unique RESTful HTTP API which allows users to programmatically access the cloud and invoke its capabilities.<br />
<br />
OpenStack can be used in both public cloud and private cloud deployment models. It enables the Infrastructure as as Service service model.<br />
<br />
All in all, OpenStack allows an individual (or a group) to convert a data center of any size into a cloud, as defined above. With OpenStack in place, the infrastructure in the data center can then be offered to tenants either publicly or privately as a service. Tenants can use the OpenStack API or Web UI to create virtual servers, establish virtual networks, manage virtual storage volumes, and much, much more.<br />
<br /></div>
Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-82814882456468944262014-07-15T02:55:00.000-04:002014-07-15T02:55:33.332-04:00CentOS 7: Install VNC ServerVNC is a remote desktop protocol, commonly used for Linux systems. The remote system runs a VNC server which can then be accessed by a local VNC client. The TigerVNC server can be installed from the CentOS-7 repository with the following yum command:<br/>
<br/>
<code># yum install tigervnc-server</code><br/>
After the TigerVNC server is installed, it can be started from the command line:<br/>
<br/>
<code># vncserver</code><br/>
If at this point, the server is still inaccessible from a VNC client, check firewall status. Note that CentOS 7 uses the firewalld service, instead of iptables.<br/>
<br/>
<code># service firewalld status</code></br>
Assuming the firewall is running, it is likely blocking the port used by the VNC server. Try manually opening this port with the following command:<br/>
<br/>
<code># firewall-cmd --permanent --add-port=5901/tcp</code><br/>
If the VNC client can connect to the server but the desktop is blank, the server is missing a desktop environment. The CentOS-7 repository includes packages for either KDE or GNOME desktop environments. KDE can be installed using the following yum command:<br/>
<br/>
<code># yum groupinstall "KDE Plasma Workspaces"</code><br/>
In order for the VNC server to make use of KDE, edit ~/.vnc/xstartup to contain the following:<br/>
<br/>
<code>
#!/bin/sh<br/>
unset SESSION_MANAGER <br/>
unset DBUS_SESSION_BUS_ADDRESS <br/>
startkde & <br/>
</code><br/>
Finally, kill and restart the VNC session:<br/>
<br/>
<code># vncserver -kill :1</code>
<code># vncserver</code></br>
Optionally, the screen resolution of the VNC session can be set with the -geometry option:<br/>
<br/>
<code># vncserver -geometry 1920x1080</code>Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-64306242264814414062014-07-15T02:35:00.000-04:002014-07-15T02:35:09.579-04:00CentOS 7: Install KDE DesktopThe K Desktop Environment (KDE) can be installed from the CentOS-7 repository using the following yum command. KDE is a Linux desktop environment.<br />
<br />
<code>yum groupinstall "KDE Plasma Workspaces"</code>
<br />
After installing KDE, it can be started from the shell (or from an xstartup script) with the following command:<br />
<br />
<code>startkde &</code>
<br/>
The above commands are also compatible with RHEL 7.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-68123917462927922422011-06-30T16:21:00.000-04:002014-07-15T02:20:41.501-04:00Generate a Random File in LinuxFor testing purposes, sometimes it's useful to generate a lot of random data, e.g. for testing the effect of file compression on a system. Generating random file data is simple using the random number generator in the /dev directory. Reading from /dev/urandom will result in a random byte. Running the following command will dump random characters to the display:<br />
<br />
<code>cat /dev/urandom</code><br />
To generate and write random data to a file, use the following command:<br />
<br />
<code>cat /dev/urandom > randfile</code><br />
To limit the output, Ctrl+C the process, or use a sleep followed by kill. For example:<br />
<br />
<code>(cat /dev/urandom > randfile &); sleep 5; killall cat</code><br />
The command sequence above will write random data to randfile for five seconds. The actual amount of data written will vary.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-85852455974468860922011-06-28T16:27:00.000-04:002014-07-15T02:25:53.726-04:00Useful .vimrc SettingsCustomizing vi can improve your productivity. By editing ~/.vimrc you can define a set of commands which are run when vi starts. The following commands can be especially useful.<br />
<br />
<code>:set showmatch</code>
highlights the matching brace or parenthesis in your code<br/><br/>
<code>:syntax on</code>
enables basic syntax highlighting<br /><br/>
<code>:set expandtab</code>
mixing tabs and spaces in source files is bad news<br />
enable expandtab to automatically convert your tabs to spaces<br /><br/>
<code>:set ts=4</code>
sets the number of spaces used to display a tab<br /><br/>
<code>:set shiftwidth=4</code>
sets the number of spaces by which to indent as a result of '>>' for example<br /><br/>
<code>:set ruler</code>
displays the current line and column number<br />
useful for restricting your code to 80 characters (if you're into that)<br /><br/>
At the very least, syntax highlighting provides a major boost to readability.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-39606625325167791832011-06-28T16:00:00.000-04:002014-07-15T02:26:39.581-04:00vi - find and replaceThe vi editor is a powerful text editor available on most unix systems. To find and replace text in vi, run the following command:<br />
<br />
<code>:%s/original/replacement/g</code>
<br />
If you'd like to be a little more careful about which instances of the original string are replaced, add the 'c' option to the end to enable checking:<br />
<br />
<code>:%s/original/replacement/gc</code>
<br />
Replacing text around a string can be accomplished with a regex:<br />
<br />
<code>:%s/abc\(.*\);/def\1;/g</code>
<br />
The command above replaces lines of the form abcblah; with defblah; preserving the blah, whatever it may be. The escaped parentheses form a regex group and can be referenced later using \1, \2, \3, etc. (the number corresponding to the group).Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-84801223887038494812011-06-20T12:16:00.000-04:002014-07-15T02:28:33.620-04:00Monitor Process Memory in LinuxTracking memory on a per-process basis can be useful for detecting memory leaks. The top utility takes memory usage information from the /proc directory and displays it in a human readable format. To measure process memory over time, use top's batch mode:<br />
<br/>
<code>top -b | grep myproc > mem_usage_file</code>
<br/>
The command above will print information for "myproc" periodically to the file "mem_usage_file". A line in the file might take the following form:<br />
<br/>
<code>29439 matt 19 0 139m 87m 10m S 9 0.5 5:39.09 myproc</code>
<br/>
The virtual memory for myproc is 139MB, the resident memory is 87MB, and the shared memory is 10MB. These values could be extracted and plotted using any scripting language.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-55507310446553863312011-06-15T14:38:00.000-04:002014-07-15T02:29:45.805-04:00Simple Python HTTP Web ServerInstalling and loading up Apache is usually overkill for hosting a simple web server. Python provides a built in HTTP server with limited functionality. In its simplest form, the following python script could be run from any directory to give access to the directory contents via HTTP:<br />
<br/>
<code>
import SimpleHTTPServer, SocketServer<br />
handler = SimpleHTTPServer.SimpleHTTPRequestHandler<br />
server = SocketServer.TCPServer(("",80), handler)<br />
server.serve_forever()<br />
</code><br />
Running this server will serve up the process' working directory content to any requesting web browser or http client. Customizing this server to provide dynamic web pages is as easy as subclassing the SimpleHTTPRequestHandler. For example:<br />
<br /><code>
import SimpleHTTPServer, SocketServer<br />
class MyHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):<br />
def do_GET(self):<br />
if self.path == "mypage.html":<br />
self.wfile.write("<html><body>") self.wfile.write("Hello World!") self.wfile.write("</body></html>")<br />
return<br />
SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)<br />
handler = MyHandler<br />
server = SocketServer.TCPServer(("",80), handler)<br />
server.serve_forever()<br />
</code><br />
The do_GET method overrides the definition in SimpleHTTPRequestHandler to implement a custom action. The script above will return a page with "Hello world!" in its body if a GET is received for "mypage.html". Any other path will result in a call to the original implementation of do_GET().Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-60803197564868653242011-06-15T14:02:00.001-04:002014-07-15T02:30:17.767-04:00Read from stdin in PythonMany unix command line utilities take input from standard input and write to standard output. This convention allows multiple commands to be piped together. Reading from standard input in python provides an easy way to build new utilities:<br />
<br /><code>
import sys<br />
for line in sys.stdin:<br />
print(line)<br />
</code><br />
The code above could be modified to accomplish a number of simple tasks, such as splitting each line by white space and only printing the first element (as an alternative to awk).Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-54006696803950846892011-06-15T10:13:00.000-04:002014-07-15T02:31:55.294-04:00Number of lines in a fileCounting the number of lines in given file is generally useful, especially in conjunction with grep.<br />
<br/>
<code>grep "abc" * -r | wc -l </code>
<br/>
<code>wc -l myfile</code>
<br/>
Word count also has the ability to count the number of words or characters in a file using the -w and -m options respectively.
Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-78463271178661594672011-03-03T15:40:00.000-05:002011-03-03T15:40:42.010-05:00Skipping incompatible library when searching, linker errorWhile compiling or linking libraries, the linker searches for matches in the library path. If a match is found based on file name, but the objects in the library are built for a different architecture than the executable being linked, the following message may be printed:<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/usr/bin/ld: skipping incompatible /lib/libmylib.a when search for -lmylib</span></blockquote> This error is only harmful if a library with the right format cannot be found. I have encountered this problem when compiling for 32-bit architectures on a 64-bit machine. The solution usually involves recompiling the library for the target architecture.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-62749729978563302412010-08-10T10:33:00.000-04:002010-08-10T10:33:25.693-04:00Resolving a hostname (DNS lookup)A hostname can be resolved to an IP address via DNS with the nslookup command:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> nslookup example.com</span><br />
<br />
The output identifies the address of the server:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> Server: 161.44.124.122</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> Address: 161.44.124.122#53</span><br />
<br />
The file /etc/resolv.conf is used to configure name servers.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-68846111589272337092010-08-09T15:38:00.001-04:002010-08-09T15:39:12.752-04:00List functions in a library, object, or executableThe nm command may be used to list the symbol table in any object file (including static libraries, shared objects, and executables). The symbol table includes global variables and public functions.<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">nm myexe</span></blockquote><blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">nm libxyz.a</span></blockquote><blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">nm libabc.so</span></blockquote>See the <a href="http://unixhelp.ed.ac.uk/CGI/man-cgi?nm">nm man page</a> for more information.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-61211205517721137982010-08-06T17:31:00.001-04:002010-08-09T12:53:07.011-04:00Executable 32-bit or 64-bitAn easy way to check if an executable is compiled for 32-bit vs. 64-bit is the file command:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">file myabc</span> </blockquote>The same command may be run on a library:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">file libxyz.so</span></blockquote>The output of the file command lists information such as whether the file is an executable or a shared object, whether or it is dynamically linked (using shared libs), etc.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-8258837311787136602010-08-06T12:00:00.000-04:002010-08-06T12:00:35.794-04:00/usr/local/libIn some Linux distros, namely Red Hat, the /usr/local/lib directory is not included in the built-in library search path. Using a shared object file in this directory may cause an error. To solve this problem, either move the library in question to one of the built-in library search directory (/usr/lib or /lib) or add /usr/local/lib to the LD_LIBRARY_PATH environment variable.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.comtag:blogger.com,1999:blog-8008778683644154864.post-63988632482412335942010-08-05T16:14:00.000-04:002010-08-05T16:14:46.162-04:00Cannot open shared object fileRan into trouble trying to run an executable the other day. Hit the following message:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">libxyz.so: cannot open shared object file: No such file or directory</span></blockquote>Made sure that the shared object file in question actually did exist:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ls /lib | grep libxyz.so</span></blockquote><blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ls /usr/lib | grep libxyz.so</span></blockquote> Turns out that the library was not in any of these standard directories, but existed in a separate directory. Made sure this separate directory was in the LD_LIBRARY_PATH:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">echo $LD_LIBRARY_PATH</span></blockquote>The directory with the library was part of the load library path, but still no luck launching the executable. The runtime linker responsible for loading up the shared objects uses cached information about the objects rather than sifting through the directory structure. The cache information can be updated by running:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ldconfig</span></blockquote>Running ldconfig crashed with a SIGBUS error. A little odd. Used the strace utility to list all the system calls made by ldconfig before the crash:<br />
<blockquote> <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">strace ldconfig</span></blockquote>The last "open" system call before the crash was to some random library I had never heard of or even used:<br />
<blockquote> <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">open("/lib/libabc.so", O_RDONLY)</span></blockquote>Running the "file" command on this mysterious library reveals some deeper problems:<br />
<blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">file /lib/libabc.so</span></blockquote>Solution: copied the same library over from another system and re-ran ldconfig.<br />
<br />
In the end, libabc.so was repaired and libxyz.so was perfectly fine. Retried the executable and it started up with no complaints.Matt Caulfieldhttp://www.blogger.com/profile/17075517865581342596noreply@blogger.com