Data Mining Practical Machine Learning Tools and Techniques

From the Book:
“Clean code that works” is Ron Jeffries’ pithy phrase. The goal is clean code that works, and for a whole bunch of reasons:
Clean code that works is a predictable way to develop. You know when you are finished, without having to worry about a long bug trail.Clean code that works gives you a chance to learn all the lessons that the code has to teach you. If you only ever slap together the first thing you think of, you never have time to think of a second, better, thing. Clean code that works improves the lives of users of our software.Clean code that works lets your teammates count on you, and you on them.Writing clean code that works feels good.But how do you get to clean code that works? Many forces drive you away from clean code, and even code that works. Without taking too much counsel of our fears, here’s what we do—drive development with automated tests, a style of development called “Test-Driven Development” (TDD for short). 
In Test-Driven Development, you:
Write new code only if you first have a failing automated test.Eliminate duplication.

Two simple rules, but they generate complex individual and group behavior. Some of the technical implications are:You must design organically, with running code providing feedback between decisionsYou must write your own tests, since you can’t wait twenty times a day for someone else to write a testYour development environment must provide rapid response to small changesYour designs must consist of many highly cohesive, loosely coupled components, just to make testing easy

The two rules imply an order to the tasks ofprogramming:
1. Red—write a little test that doesn’t work, perhaps doesn’t even compile at first
2. Green—make the test work quickly, committing whatever sins necessary in the process
3. Refactor—eliminate all the duplication created in just getting the test to work


Red/green/refactor. The TDD’s mantra.

Assuming for the moment that such a style is possible, it might be possible to dramatically reduce the defect density of code and make the subject of work crystal clear to all involved. If so, writing only code demanded by failing tests also has social implications:
If the defect density can be reduced enough, QA can shift from reactive to pro-active workIf the number of nasty surprises can be reduced enough, project managers can estimate accurately enough to involve real customers in daily developmentIf the topics of technical conversations can be made clear enough, programmers can work in minute-by-minute collaboration instead of daily or weekly collaborationAgain, if the defect density can be reduced enough, we can have shippable software with new functionality every day, leading to new business relationships with customers


So, the concept is simple, but what’s my motivation? Why would a programmer take on the additional work of writing automated tests? Why would a programmer work in tiny little steps when their mind is capable of great soaring swoops of design? Courage.
Courage
Test-driven development is a way of managing fear during programming. I don’t mean fear in a bad way, pow widdle prwogwammew needs a pacifiew, but fear in the legitimate, this-is-a-hard-problem-and-I-can’t-see-the-end-from-the-beginning sense. If pain is nature’s way of saying “Stop!”, fear is nature’s way of saying “Be careful.” Being careful is good, but fear has a host of other effects:
Makes you tentativeMakes you want to communicate lessMakes you shy from feedbackMakes you grumpy

None of these effects are helpful when programming, especially when programming something hard. So, how can you face a difficult situation and:
Instead of being tentative, begin learning concretely as quickly as possible.Instead of clamming up, communicate more clearly.Instead of avoiding feedback, search out helpful, concrete feedback.(You’ll have to work on grumpiness on your own.) 

Imagine programming as turning a crank to pull a bucket of water from a well. When the bucket is small, a free-spinning crank is fine. When the bucket is big and full of water, you’re going to get tired before the bucket is all the way up. You need a ratchet mechanism to enable you to rest between bouts of cranking. The heavier the bucket, the closer the teeth need to be on the ratchet.

The tests in test-driven development are the teeth of the ratchet. Once you get one test working, you know it is working, now and forever. You are one step closer to having everything working than you were when the test was broken. Now get the next one working, and the next, and the next. By analogy, the tougher the programming problem, the less ground should be covered by each test.

Readers of Extreme Programming Explained will notice a difference in tone between XP and TDD. TDD isn’t an absolute like Extreme Programming. XP says, “Here are things you must be able to do to be prepared to evolve further.” TDD is a little fuzzier. TDD is an awareness of the gap between decision and feedback during programming, and techniques to control that gap. “What if I do a paper design for a week, then test-drive the code? Is that TDD?” Sure, it’s TDD. You were aware of the gap between decision and feedback and you controlled the gap deliberately.

That said, most people who learn TDD find their programming practice changed for good. “Test Infected” is the phrase Erich Gamma coined to describe this shift. You might find yourself writing more tests earlier, and working in smaller steps than you ever dreamed would be sensible. On the other hand, some programmers learn TDD and go back to their earlier practices, reserving TDD for special occasions when ordinary programming isn’t making progress.

There are certainly programming tasks that can’t be driven solely by tests (or at least, not yet). Security software and concurrency, for example, are two topics where TDD is not sufficient to mechanically demonstrate that the goals of the software have been met. Security relies on essentially defect-free code, true, but also on human judgement about the methods used to secure the software. Subtle concurrency problems can’t be reliably duplicated by running the code.

Once you are finished reading this book, you should be ready to:
Start simplyWrite automated testsRefactor to add design decisions one at a time

This book is organized into three sections.
An example of writing typical model code using TDD. The example is one I got from Ward Cunningham years ago, and have used many times since, multi-currency arithmetic. In it you will learn to write tests before code and grow a design organically.An example of testing more complicated logic, including reflection and exceptions, by developing a framework for automated testing. This example also serves to introduce you to the xUnit architecture that is at the heart of many programmer-oriented testing tools. In the second example you will learn to work in even smaller steps than in the first example, including the kind of self-referential hooha beloved of computer scientists.Patterns for TDD. Included are patterns for the deciding what tests to write, how to write tests using xUnit, and a greatest hits selection of the design patterns and refactorings used in the examples.

I wrote the examples imagining a pair programming session. If you like looking at the map before wandering around, you may want to go straight to the patterns in Section 3 and use the examples as illustrations. If you prefer just wandering around and then looking at the map to see where you’ve been, try reading the examples through and refering to the patterns when you want more detail about a technique, then using the patterns as a reference.

Several reviewers have commented they got the most out of the examples when they started up a programming environment and entered the code and ran the tests as they read.

A note about the examples. Both examples, multi-currency calculation and a testing framework, appear simple. There are (and I have seen) complicated, ugly, messy ways of solving the same problems. I could have chosen one of those complicated, ugly, messy solutions to give the book an air of “reality.” However, my goal, and I hope your goal, is to write clean code that works. Before teeing off on the examples as being too simple, spend 15 seconds imagining a programming world in which all code was this clear and direct, where there were no complicated solutions, only apparently complicated problems begging for careful thought. TDD is a practice that can help you lead yourself to exactly that careful thought.

Test Driven Development: By Example

Nowadays, numerous journals and conferences have published articles related to context-aware systems, indicating many researchers' interest. Therefore, the goal of this paper is to review the works that were published in journals, suggest a new classification framework of context-aware systems, and explore each feature of classification framework. This paper is based on a literature review of context-aware systems from 2000 to 2007 using a keyword index and article title search. The classification framework is developed based on the architecture of context-aware systems, which consists of the following five layers: concept and research layer, network layer, middleware layer, application layer and user infrastructure layer. The articles are categorized based on the classification framework. This paper allows researchers to extract several lessons learned that are important for the implementation of context-aware systems.

Context-aware systems

Our goal is to use the vast repositories of available open source code to generate specific functions or classes that meet a user's specifications. The key words here are specifications and generate. We let users specify what they are looking for as precisely as possible using keywords, class or method signatures, test cases, contracts, and security constraints. Our system then uses an open set of program transformations to map retrieved code into what the user asked for. This approach is implemented in a prototype system for Java with a web interface.

/pdf/semantics-based-code-search-4bk4pqywze.pdf

Semantics-based code search

Different studies show that programmers are more interested in finding definitions of functions and their uses than variables, statements, or arbitrary code fragments [30, 29, 31]. Therefore, programmers require support in finding relevant functions and determining how those functions are used. Unfortunately, existing code search engines do not provide enough of this support to developers, thus reducing the effectiveness of code reuse. We provide this support to programmers in a code search system called Portfolio that retrieves and visualizes relevant functions and their usages. We have built Portfolio using a combination of models that address surfing behavior of programmer and sharing Related concepts among functions. We conducted an experiment with 49 professional programmers to compare Portfolio to Google Code Search and Koders using a standard methodology. The results show with strong statistical significance that users find more relevant functions with higher precision with Portfolio than with Google Code Search and Koders.

/pdf/portfolio-finding-relevant-functions-and-their-usage-1jjiek5m6a.pdf

Portfolio: finding relevant functions and their usage

We present CodeGenie, a tool that implements a test-driven approachto search and reuse of code available on large-scale coderepositories. While using CodeGenie developers design test cases fora desired feature first, similar to Test-driven Development (TDD).However, instead of implementing the feature as in TDD, CodeGenieautomatically searches for it based on information available in thetests. To check the suitability of the candidate results in thelocal context, each result is automatically woven into thedeveloper's project and tested using the original tests. Thedeveloper can then reuse the most suitable result. Later, reusedcode can also be unwoven from the project as wished. For the codesearching and wrapping facilities, CodeGenie relies on Sourcerer, anInternet-scale source code infrastructure that we have developed

CodeGenie: using test-cases to search and reuse source code

Complex and expensive testing and evaluation can inhibit cost-effective development of ubiquitous computing applications. A new environment for testing and evaluating system- and network-related issues in location-based applications strives to evaluate these applications prior to deployment. We discuss a health-monitoring application and its simulation environment.

Evaluating a location-based application: a hybrid test and simulation environment

Caller-REP: Detecting unwanted calls with caller social strength

Spam in telephony refers to unsolicited, automatically dialed, pre-recorded phone calls often used for marketing or phishing. Spam in telephony can affect the productivity of human beings and deprive legitimate users of phone network resources. Spam over Internet Telephony (SPIT) is a challenge and can become a major threat when telephony completely shifts to VoIP. Email anti-Spam techniques are typically not applicable to SPIT detection: SPIT detection needs to be near real-time; call content is not available for processing before the call is established; there are only a small number of features available for examination; and signaling messages of spammer and non-spammer are similar. SPIT detection in transit VoIP operator is a non-trivial task because of the high call volume and the unavailability of direct interaction with a caller or a callee. In this paper we present a multistage SPIT detection framework that addresses these issues. Our framework blocks SPIT based on the social relationships and statistical features of the caller. The framework consists of three modules: a blacklist, a statistical module, and a trust module. For achieving better results and resource optimization, we provide feedback on where these modules should be placed in the transit network. Our experimental results show that the detection response time increases slightly after system deployment, but this increase is not noticeable to the callers. Besides SPIT detection the framework can also detect flooding, vishing and number scan attacks.

Multistage SPIT detection in transit VoIP

Middleware is being extensively used in Internet of Things (IoT) deployments and is available in a variety of flavors — from general-purpose community-driven middleware and telco-developed Machine-to-Machine (M2M) middleware to middleware targeting specific deployments. Despite this extensive use and diversity, little is known about the benefits, disadvantages, and performance of each middleware platform and how the different platforms compare with each other. This comparison is especially relevant to help the design and dimensioning of IoT infrastructure. In this paper, we propose a set of qualitative dimensions and quantitative metrics that can be used for bench-marking IoT middleware. We use the publication-subscription of a large dataset as use case inspired by a smart city scenario to compare two middleware platforms. The methodology enables us to systematically compare the two middleware platforms. Further, we are able to use our approach to identify inefficiencies in implementations and to characterize performance variations throughout the day, showing that the metrics may also be used for monitoring.

Ricardo Morla

Papers

CodeGenie: using test-cases to search and reuse source code

Evaluating a location-based application: a hybrid test and simulation environment

Caller-REP: Detecting unwanted calls with caller social strength

Multistage SPIT detection in transit VoIP

Benchmarking IoT middleware platforms