Elements of Dance Etiquette (2005)

from Hacker News https://ift.tt/YV9WJO
via IFTTT

Elements of Dance Etiquette

Aria Nosratinia

Also see Beyond Dance Etiquette: Success and Enjoyment in Social Dancing

Dance etiquette is a set of guidelines that help us navigate the social dimensions of dancing.

Why do we care about dance etiquette? Because it is nice to know how to go about in the dancing circles. It makes the difference between having a happy or unhappy dancing experience, the difference between people wanting, or not wanting to dance with you.

Dancing has its own culture. If you want to join a group of dancers and enjoy their company, it is a good idea to follow the accepted costums of their dance group. One of the ways you get accepted into a group is by the way you're dressed.

The more formal the dance, the more formal the outfit. For example, if you are invited to a formal charity ball, anything less than a tuxedo for men or ball gown for women would be inappropriate. On the other hand, at a dance lesson at your local studio, there is usually no need to dress formally.

This is not as hard as it may seem; a little common sense goes a long way. Also, if in doubt, follow the crowd! See what others do and follow suit. If all else fails, you can always ask the dance organizers about the dress code.

Below I give a guideline and explanation for dress code, which you may see on invitations and announcements, as well as a general idea of what to wear at different dance venues.

White tie: White tie is the most formal category of dressing. For the gentleman, it means a black tailcoat with matching trousers trimmed by ribbon of braid or satin on the outside of each trouser leg, a white pique' tie, white pique' single or double-breasted vest, and a wing-collar shirt with a stiff pique' front. White gloves are nice optional accessories for gentlemen. The lady appears in a ball gown, which is an evening dress with a full skirt, possibly with open back and low neck line. Elbow-length gloves are a nice addition for the lady.
Black tie: Gentlemen in black tuxedo coat, trousers trimmed with satin ribbon along the outside of the legs, cummerband and bow tie. The phrase ``black tie'' does not refer to the color of the tie. In fact colorful ties (with matching cummerbands) are very popular. Ladies appear in ball gowns.
Black tie optional: Same as above, except gentlemen have the option of wearing a regular suit with a tie (bow tie preferred), and ladies wear a cocktail gown or dinner dress. Long to full-length skirts are preferred; short skirts are not recommended.
Formal: Gentlemen in suit and tie (nowadays a sport coat is often an acceptable replacement for a full suit), ladies in cocktail gown or evening dress.
Semi-formal: Gentlemen in dress slacks with dress shirt and tie, jacket is optional. Other options include a vest or a sweater that shows the tie. At the lower end of formality, these events can be attended without a tie, e.g. with a turtleneck and jacket. Ladies in evening dress or dinner dress, but other chic outfits are also acceptable (like flowing pants, etc.)
Dressy Casual: Applies to most practice dances, workshops, and dance lessons. Gentlemen can wear coton slacks with solid color T-shirt, turtleneck, mock turtleneck, or polo shirt. Ladies have a much wider set of clothing options. Use your imagination and sense of fashion. In general this is a conservative and toned-down appearance that has grown increasingly popular on the dance floors. Don't forget your dance shoes!
Country/Western: Country western attire has variations across the country, but generally it is acceptable to go in blue or black jeans (not stone-washed) and cowboy boots. Make sure that the boots will not mark the dance floor. If you wear a hat, it may be a good idea to take it off when going on the floor. Note that country western folks can be very sensitive about their hats. It is improper to touch or otherwise handle someone's hat, even if it sits on a table. For a lady to pick up and put on a gentleman's hat is considered very flirtatious.
Milongas: (Argentine Tango) For both ladies and gentlemen, black or dark themes are preferred.
Latin: This refers to venues that specialize in Salsa, Merengue, Cumbia, etc. For gentlemen, any button-up shirt, solid T-shirt or mock turtleneck, dress slacks, and dance shoes. Jackets are nice, but a vest can be even more stylish. Unlike most other dance venues, bright and colorful outfits for gentlemen are acceptable, although dark themes are more common. Ladies can (and often do) wear sexy outfits: both short skirts and longer slit skirts are popular. Low necklines and exposed midriffs are not uncommon.
Swing: There are no strict rules for swing outfits. Both the Gentleman and the Lady wear outfits that are reasonably neat and chic, although often not very formal. Many types of swing are fast-paced and athletic, so wearing suitable clothing is essential. For example, the Lady would be well advised to stay away from short, tight skirts. See also the next section on Comfort and Safety. A cute trend, especially in Lindy Hop circles, is to wear vintage outfits from the 1930's and 40's. But this is not done everywhere and is not at all a requirement.

Wear clothing that makes it easy and enjoyable to dance, both for yourself and your partner.

Regardless of how informal the dance is, always wear dance shoes. Do not wear sneakers or other shoes with rubber or spongy soles. They can stick to the floor during turns and spins and cause ankle and knee injuries.
Avoid sleeveless shirts and strapped dresses, especially for active dancing: It is not pleasant to have to touch the damp skin of a partner.
Sleeves that are baggy or cut low in the armpit are not a good idea, especially in Latin and swing dancing, because dancers need access to partner's back, and hands may get caught in baggy sleeves.
Accessories like big rings, watches, brooches, loose/long necklaces, and big belt buckles can be dangerous. They can catch in partner's clothing, scratch and bruise.
Gentlemen: if you have no place to leave your keys and loose change, carry them in the *left* pocket of your trousers. This makes it less likely to bruise your partner.
Long hair should be put up or tied in a pony tail. It is difficult to get into closed dance position when the lady has long flowing hair (hair gets caught in gentleman's right hand). It is also not fun to be hit in the face with flying hair during turns and spins.

Dancing is an activity where two people come in close contact. Before a dance:

Shower and use a deodorant,
Brush teeth and use mouthwash or breath mint,
Abstain from foods that produce strong odors, like those heavy in garlic
The odor of cigarettes on one's breath or clothing can be very unattractive.

During a dance:

Check your grooming periodically
During active dance sessions, freshen up and towel off periodically in the bathroom
Gentlemen, you can carry an extra shirt with you to the dance, in case you need a change.

When asking for a dance, it is easiest to stay with traditional phrases:

``May I have this dance?''
``May I have this Waltz/Rumba/Foxtrot/etc.''
``Would you like to dance?''
``Care to dance?''
``Shall we dance?''

In the past it has been the tradition that men asked women to dance. But this custom has gradually changed. Today, women should feel equally comfortable asking a partner for a dance, even in a formal setting.

If your desired partner is with a group, be unambiguous and make eye contact when asking for a dance. If you vaguely approach a group, two individuals may think you are asking for a dance. You can imagine that the one not getting the dance is going to be miffed. Let's avoid such awkward moments by a decisive approach and solid eye contact.

What if you want to ask someone to dance, who is enganged at the moment in a conversation? Is it acceptable to interrupt a conversation to ask someone to dance? Some would say that one's presence in a dancing establishment indicates a desire for dancing and everyone is fair game. Others say that interrupting a conversation is rude.

In my opinion, ask someone to dance if you think he/she is ready to dance and will enjoy dancing with you at that moment. This requires you to be a good judge of the moment. Also, if you know someone well enough to know they don't mind being interrupted, then go ahead and ask them.

Perhaps one way to handle this is to walk gently to the edge of your intended partner's "personal space", which is about 3-4 feet (one meter). It will give you an opportunity to ask them to dance. If your presence is not acknowledged, then it may be a good idea to find someone else for that dance.

Exercising common sense and social skills is always a good idea. If someone is sitting closely with their significant other, whispering sweet nothings to each other, then it is probably not a good time to ask either of them for a dance. Now a different scenario: your intended partner is cornered by a bore and being lectured on weather patterns in lower Namibia. You can advance and stand close. Once your intended partner makes eye contact with you, smile and say: ``Dance?'' Usually, that is enough to do the job. If not, it is better to leave him/her to learn about weather patterns in lower Namibia.

Sometimes two individuals simultaneously ask someone for a dance. In that situation, dance etiquette recommends that the object of attention should accept one of the dances, while offering a later dance to the other one.

If each person dances with only one or two others, the social dynamics of dancing will be compromised. For that reason, dance etiquette strongly encourages everyone to dance with many different partners. This is to ensure a diversity of partnerships on the floor, and to give everyone a chance to dance. Specifically, dance etiquette rules against asking the same partner for more than two consecutive dances.

One of the common violations of this rule occurs when someone dances most of the night with their escort. The ruling of etiquette in this case is much the same as for the traditional (formal) dinner parties: one never sits down to dinner next to one's spouse. It is assumed that if spouses were interested primarily in talking with one another, they could have stayed home together. By the same token, going to a social dance demonstrates a desire to dance socially. This means dancing with a host of partners, and not just with one or a select few. I have heard a version of this rule that reserves the first and last dance of the evening to be done with one's escort, and other dances with others.

People generally tend to dance with others at their own level, but you should try to dance socially with partners of all levels. Dance etiquette frowns disapprovingly on those who only dance with the best dancers on the floor. Although this is not a terrible offense, it is still bad form. Better dancers are especially advised to ask beginners to dance. Not only does this help the social dynamics of a dance, it also helps the better dancer (although it is outside the scope of this discussion to explain why or how).

Unfortunately, there are some social dancers who consider themselves too good to dance with beginners, who cannot ``keep up'' with their level of dancing. It is often the case that these dancers are not as good as they think. They need good partners because only good partners can compensate for their mistakes, bad technique, or other inadequacies. The truly good dancers often seek the challenge of dancing with those at lower levels, and enjoy it. Good dancers make their partners look good.

Being declined is always unpleasant. For beginners and shy individuals it is even harder to take, and may discourage them from social dancing. Dance etiquette requires that one should avoid declining a dance under most circumstances. For example, there is no correct way of refusing an invitation on the basis of preferring to dance with someone else. According to tradition, the only graceful way of declining a dance is either (a) you do not know the dance, (b) you need to take a rest, or (c) you have promised the dance to someone else.

The last excuse should be used only sparingly. When declining a dance, it is good form to offer another dance instead: ``No, thank you, I'm taking a break. Would you like to do another dance later?'' Also, declining a dance means sitting out the whole song. It is inconsiderate and outright rude to dance a song with anyone after you have declined to dance it with someone else. If you are asked to dance a song before you can ask (or get asked by) your desired partner, that's the luck of the draw. The choices are to dance it with whomever asked first, or to sit out the dance.

Does dance etiquette allow declining a dance outside of the cases mentioned above? The answer is yes, if someone is trying to monopolize you on the dance floor, make inappropriate advances, is unsafe (e.g. collides with others on the floor), or is in other ways unsavory, you are within the bounds of etiquette to politely but firmly decline any more dances. Perhaps the simplest, best way is to say ``No, thank you,'' without further explanation or argument. Dancers are encouraged to use discretion and restraint when exercising this option.

The first thing to do when one is turned down for a dance is to take the excuse at face value. Typical social dance sessions can be as long as three to four hours, and there are few dancers who have the stamina of dancing non-stop. Everyone has to take a break once in a while, and that means possibly turning down one or two people each time one takes a break. The advice to shy dancers and especially beginners is not to get discouraged if they are turned down once or twice.

However, since social dancers are generally nice and polite, being repeatedly declined can be a signal. In that case, it is a good idea to examine one's dancing and social interactions to see if anything is wrong.

The dancing on a floor is done along a counter clockwise direction, known as the Line Of Dance. This applies to traveling dances including Waltz, Foxtrot, Tango, Quickstep, and Viennese Waltz, as well as Polka and two-step in the country western repertoire. Latin and Swing dances are more or less stationary and have no line of dance. Sometimes it is possible to dance more than one type of dance to the same song. For example, some Foxtrots can also be swings, and many Lindy Hop songs are just great for Quickstep. In that case, swing dancers take the middle of the floor, and the moving dancers move along the periphery in the direction of the line of dance. Some caution should be exercised when getting on the dance floor, especially if the song has already started and couples are dancing on the floor. It is the responsibility of incoming couples to make sure that they stay out of the way of the couples already dancing. Specifically, before getting into dance position, one should always look opposite the line of dance to avoid blocking someone's way, or even worse, causing a collision. After the dance is finished and before parting, thank your partner. This reminds me of a social partner who, upon being thanked at the end of the dance, would answer: ``You're welcome!'' This always gave me a funny feeling. The proper answer to ``Thank you!'' on the dance floor is: ``Thank you!'' The point is that the thanks is not due to a favor, but to politeness.

If you enjoyed the dance, let your partner know. Compliment your partner on her/his dancing. Be generous, even if he/she is not the greatest of dancers. Be specific about it if you can: ``I really enjoyed that double reverse spin. You led/followed that beautifully!'' If you enjoyed it so much that you would like to have another dance with him/her again, this is a good time to mention it: ``This Waltz went really great! I'd like to try a Cha-Cha with you later.'' Although remember that dancing too many dances with the same partner and booking many dances ahead are both violations of social dance rules.

When a song comes to an end, leave the floor as quickly as it is gracefully possible. Tradition requires that the gentleman give his arm to the lady and take her back to her seat at the end of the dance. While this custom is linked to the outdated tradition requiring the gentlemen to ask ladies for dances, it is still a nice touch, although it may be impractical on the more crowded dance floors. In any case, remember that your partner may want to get the next dance. Don't keep them talking after the dance is over, if they seem ready to break away to look for their next partner. Some dance floors, especially in country western dance establishments, have limited access space (most of the periphery is railed). Dancers and onlookers should avoid blocking these entrances. In particular, avoid stopping to chat immediately after exiting the dance floor. Another issue in Country Western dancing regards line dancers, who sometimes share the floor with other dancers. They should avoid blocking entrances from the inside while dancing. Responsible usage of the floor requires that one stays out of the way of others. Some figures require a momentary movement against line of dance. These figures should be executed with great caution on a social dance floor, and only when there is no danger of collision. Avoid getting too close to other couples, especially less experienced ones. Be prepared to change the directions of your patterns to avoid congested areas. This requires thinking ahead and matching your patterns to the free areas on the floor (floorcraft). While this may sound complicated to the novice dancer, it gradually becomes second nature.

Sharing the floor sometimes means leaving the floor! For example, if there are too many dancers to fit on the floor, then a considerate dancer would withdraw every few dances to let everyone dance. The same idea applies if there aren't the same number of men and women. Then there is a mismatch and for each song some people will be left without a partner. If there aren't enough partners, it would be nice to voluntarily withdraw every few dances so that everyone gets a chance to dance.

Another aspect of sharing the floor is to match one's speed to that of others. In a recent social dance, a particularly tall and handsome couple caught my eye. They were moving with great speed and skill across the floor, and I began to enjoy watching them dance. But then I noticed they were coming dangerously close to other dancers on the crowded dance floor, and many times other couples came to a stop and moved out of their way. It was easy to see they were unhappy about this couple ``taking over'' the floor.

The only thing to be said about aerials on the social dance floor is: don't do them. While they may look ``cool,'' the execution of aerials requires training by a qualified instructor. Don't do them by yourself unless you are trained, and certainly don't do them on the social dance floor. Dancers have been badly hurt by either participating in aerials, or unluckily being in the proximity of those who did. In fact, in 1996, a swing dancer died during the execution of an aerial. Aerials can be extremely dangerous, please take this issue seriously.

The same principle applies to other lifts and drops, as well as choreographed patterns that require a large amount of floor space.

Never blame a partner for missed execution of figures. Once in a social dance I accidentally overheard a novice couple, where the lady said: ``I can do this step with everyone but you!'' The fact that she was wrong (I had seen her other attempts) is irrelevant. The point is that she was unkind and out of line. Even if the gentleman were at fault, she was not to say something like that (more about this in the section: ``dancing to the level of partner.'')

Regardless of who is at fault when a dancing mishap occurs, both parties are supposed to smile and go on. This applies to the better dancer in particular, who bears a greater responsibility. Accepting the blame is especially a nice touch for the gentleman. But at the same time, do not apologize profusely. There is no time for it, and it makes your partner uncomfortable.

My personal preference is the following: whenever something untoward happens, I first see if my partner noticed. Sometimes the partner may not be aware, for example, that a figure was slightly off-time or that a fine point in technique was missed, in which case it is better to let it go. If she has noticed, I just smile and whisper ``sorry...'' and go on, regardless of whose fault it was.

It often happens that the two partners dancing socially are not at the same level. It is important that the more experienced partner dances at the level of the less experienced partner. This is mostly a comment for leaders: when dancing with a new partner, start with simple figures, and gradually work your way up to more complicated patterns. You will discover a comfort level, file it away in memory for the next time you dance with the same partner.

The same principle applies to Latin and Swing followers, although to a lesser degree. Doing extra syncopations, footwork, free spins etc. can be distracting and even intimidating for a less experienced leader. Although I must say that the show-off follower is rather rare; most of the violations of this sort are by leaders who lead inexperienced partners into complicated figures.

Social dancers strive to make their partners comfortable and help them enjoy the dance. This requires sensitivity to the likes and dislikes of the partner. These preferences can take a variety of forms. For example, I remember that one of my West Coast Swing social partners found neck wraps uncomfortable. In the same manner, some dancers don't like spins (or many spins in a row), while others really enjoy them. Some like extended syncopations and others don't. There are many more examples in various dance venues. Be sensitive to your partners. It is not too hard to detect their likes and dislikes, and if in doubt, ask. Be personable, smile, and make eye contact with your partner. Try to project a warm and positive image on the dance floor, even if that is not your personal style. Many of us lead hectic lives that include a difficult balance between study, work, family, and other obligations. Having a difficult and tiring day, however, is not an acceptable excuse for a depressing or otherwise unpleasant demeanor on the dance floor. Because of the setting of a social dance, we do not always dance with our favorite partners. This is also not grounds for a cold treatment of the partner. Once one asks or accepts a dance, it is important to be outwardly positive, even if not feeling exactly enthusiastic.

The social dancer is also well advised to be watchful of an unchecked ego. While a healthy sense of self is helpful in all social interactions, it is more attractive when mixed with an equal dose of modesty. Don't let perceived dancing abilities or physical attractiveness go to your head. It is helpful to remember that overestimating one's dance prowess or attractiveness is quite common.

There are two aspects to this point of etiquette: This is unfortunately one of the more common breaches of dance etiquette. This often happens when a dancer stops in the middle of a song to correct his or her partner, or tell them how to execute a dance figure. Ironically, this error is often committed by individuals who are not fit to teach! Experienced social dancers dance at the level of their partners. Even for experienced dancers, the social dance floor is not the place to teach or to correct your partner. It is better to concentrate on patterns that both partners can do and enjoy. Unsolicited teaching can be humiliating and takes the fun out of dancing. This is not necessarily a flagrant violation. For many, it is flattering to be consulted about a point of dancing. However, a little care and caution is always a good idea. Consider this hypothetical scenario: A polite dancer is excited when his favorite song comes on, and he asks the closest stranger for the dance. He really wants to dance this song, but she replies: ``I have never done this dance before. Can you please teach me?''

It is debatable how much one can learn, from scratch, in the 2-3 minutes a typical song plays, but that is beside the point. This is a song he really wants to dance to. For this or any other reason, he may not wish to spend time at that moment teaching someone, but she has left him no polite way of getting out. In this situation: (a) She doesn't know him (so cannot justify the imposition based on friendship), (b) she solicits teaching at the time he is asking her to dance, which puts him at a disadvantage, and (c) she does not know anything about the dance, so he cannot say: ``let's just do basic steps.''

Of course it's not always that bad. Dancers can learn quite a bit from each other in social dancing; observing a few simple points will make things enjoyable for all:

Don't say "teach me" the moment someone asks you to dance. If they are shy, they will feel trapped, will spend the next few minutes with you, and then for the rest of the night will avoid you like the plague. If they are not so shy, they will not teach you, and for the rest of the night will avoid you like the plague.
A good approach is the following: when asked to dance, one can say ``I would like to, but I don't know the dance.'' This shows that help would be appreciated, but without any pressure.
The asker in this situation can either offer to take the partner on the floor and do some basic steps, or if s/he is not so inclined, take it as a decline of dance: ``Oh, it would have been fun, perhaps we can do a different dance later?''
It is better to request help from friends, or at least someone you have had a dance or two with already, rather than someone you just met. If anythings, this is a great motivation to make friends in the dance community.
If you want to get pointers from someone, wait until s/he sits out a dance. Then go talk to her/him. This way they are not missing out on a dance by helping you.

Etiquette is here to ensure everyone has a good time in a social dance setting, so pay attention to it.
Your outfit and accessories should be comfortable, safe, and also reflect the culture and level of formality of the dance group. Most importantly, do not forget your dance shoes.
Ask everyone to dance. Do not monopolize one partner for the whole night.
Today's beginners will be the good dancers of tomorrow, so be nice to them and dance with them.
Do not decline a dance unless you absolutely have to. Having declined a dance, you cannot dance the same song with someone else.
Be considerate of other couples on the floor. Exercise good floorcraft. Do not cut other couples off. No aerials or choreographed steps on the social dance floor!
Stationary dancers (e.g. Swing dancers) stay in the middle, traveling dancers move on the boundary along the line of dance.
Avoid patterns that your partner cannot do: dance to the level of your partner.
Never blame your partner for missteps.
No unsolicited teaching on the floor!
Smile, be warm, be personable, be nice.

If you enjoyed this article, you may also wish to read Beyond Dance Etiquette: Success and Enjoyment in Social Dancing

Last modified 19 March 2005

Back to Aria's Dance Page

Back to Aria's Home Page

These pages have been accessed times since 2/1/96

Permission is granted to make and distribute printed copies of this article non-commercially. The author reserves the right to electronic versions of this article, and non-electronic copyrights are granted on the condition that the article is reproduced in its entirety and without any alterations, including this copyright notice.

Ask HN: Where should I start as a 34-year-old switching to software as a career?

https://ift.tt/2Jax2NA

Write code for microbiology? Play with computational chemistry on GPUs?

A friend at work moved from lab rat to data analysis and experiment design, switching departments from Adjuvant Research to IT. A couple of years later, I left the company to go from sysadmin to software and tech writing, and it was tough for about nine months. He went from scientific software to security research at a new company.

We were able to jump because we had few family or financial commitments. So I have no idea if what I experienced is relevant at all. But if you feel pulled in the software direction, start writing something, probably Jupyter notebook stuff. Explain Like I'm Five years old what you have experienced already.

Going through the process of sitting down (or walking, then sitting) and writing something takes a bit of focus, perhaps. I wrote a book, then wrote software. I can't recommend writing a book without a real job (my book netted me about $0.25 per hour). But write something you show other people.

from Hacker News https://ift.tt/YV9WJO
via IFTTT

Let an algorithm tell you how to eat

https://ift.tt/2EGDojv

Opinion | The A.I. Diet - The New York Times

CreditErik Blad

By Eric Topol

Dr. Topol is a cardiologist.

March 2, 2019

Some months ago, I participated in a two-week experiment that involved using a smartphone app to track every morsel of food I ate, every beverage I drank and every medication I took, as well as how much I slept and exercised. I wore a sensor that monitored my blood-glucose levels, and I sent in a sample of my stool for an assessment of my gut microbiome. All of my data, amassed with similar input from more than a thousand other people, was analyzed by artificial intelligence to create a personalized diet algorithm. The point was to find out what kind of food I should be eating to live a longer and healthier life.

The results? In the sweets category: Cheesecake was given an A grade, but whole-wheat fig bars were a C -. In fruits: Strawberries were an A+ for me, but grapefruit a C. In legumes: Mixed nuts were an A+, but veggie burgers a C. Needless to say, it didn’t match what I thought I knew about healthy eating.

It turns out, despite decades of diet fads and government-issued food pyramids, we know surprisingly little about the science of nutrition. It is very hard to do high-quality randomized trials: They require people to adhere to a diet for years before there can be any assessment of significant health outcomes. The largest ever — which found that the “Mediterranean diet” lowered the risk for heart attacks and strokes — had to be retracted and republished with softened conclusions. Most studies are observational, relying on food diaries or the shaky memories of participants. There are many such studies, with over a hundred thousand people assessed for carbohydrate consumption, or fiber, salt or artificial sweeteners, and the best we can say is that there might be an association, not anything about cause and effect. Perhaps not surprisingly, these studies have serially contradicted one another. Meanwhile, the field has been undermined by the food industry, which tries to exert influence over the research it funds.

Now the central flaw in the whole premise is becoming clear: the idea that there is one optimal diet for all people.

Only recently, with the ability to analyze large data sets using artificial intelligence, have we learned how simplistic and naïve the assumption of a universal diet is. It is both biologically and physiologically implausible: It contradicts the remarkable heterogeneity of human metabolism, microbiome and environment, to name just a few of the dimensions that make each of us unique. A good diet, it turns out, has to be individualized.

We’re still a long way from knowing what this means in practice, however. A number of companies have been marketing “nutrigenomics,” or the idea that a DNA test can provide guidance for what foods you should eat. For a fee, they’ll sample your saliva and provide a rudimentary panel of some of the letters of your genome, but they don’t have the data to back their theory up.

Coming up with a truly personalized diet would require crunching billions of pieces of data about each person. In addition to analyzing the 40 trillion bacteria from about 1,000 species that reside in our guts, as the project I participated in did, it would need to take into account all of the aspects of that person’s health, including lifestyle, family history, medical conditions, immune system, anatomy, physiology, medications and environment. This would require developing an artificial intelligence more sophisticated than anything yet on the market.

The first major development in this field occurred a few years ago when Eran Segal, Eran Elinav and their colleagues at the Weizmann Institute of Science in Israel published in the journal Cell a landmark paper titled “Personalized Nutrition by Prediction of Glycemic Responses.”

Spikes in blood-glucose levels in response to eating are thought to be an indicator of diabetes risk, although we don’t know yet if avoiding them changes that risk. These spikes are only one signature for our individualized response to food. But they represent the first objective proof that we do indeed respond quite differently to eating the same foods in the same amounts.

The study included 800 people without diabetes. The data for each person included the time of each meal, food and beverage amount and content, physical activity, height, weight and sleep. The participants had their blood and gut microbiome inhabitants assessed and their blood glucose monitored for a week. They ate more than 5,000 standardized meals provided by the researchers, which contained popular items like chocolate and ice cream, as well as nearly 47,000 meals that consisted of their usual food intake. In total, there were more than 1.5 million glucose measurements made. That’s a big data set.

Image

CreditErik Blad

Using machine learning, a subtype of artificial intelligence, the billions of data points were analyzed to see what drove the glucose response to specific foods for each individual. In that way, an algorithm was built without the biases of the scientists.

More than a hundred factors were found to be involved in glycemic response, but notably food wasn’t the key determinant. Instead it was the gut bacteria. Here were two simultaneous firsts in nutritional science: one, the discovery that our gut microbiome plays such a big role in our unique response to food intake, and the other that this discovery was made possible by A.I. The journal ran an accompanying editorial titled “Siri, What Should I Eat?”

Several subsequent studies by these researchers and others have confirmed not only our microbiome’s importance but also that a substantial proportion of healthy people have high glucose levels after eating. My curiosity about this led me to approach Dr. Segal and Dr. Elinav to ask if they would test me.

A few weeks later, my data had been ingested by their machine-learning algorithm. It turned out that my gut microbiome was densely populated by one particular bugger — Bacteroides stercoris, accounting for 27 percent of my co-inhabitants (compared with its average of less than 2 percent in the general population). I had several glucose spikes as high as 160 milligrams per deciliter of blood (normal fasting glucose levels are less than 100, but we don’t yet know what level is normal after eating).

I was then provided with a set of specific food recommendations in order to avoid glucose spikes, including that information on cheesecake and mixed nuts, and a searchable database of glucose predictions for 100,000 foods and beverages.

That sounds great, but I realized I had a big problem. For the most part the highly recommended foods, like cheese danishes, were ones I really disliked, while those rated C-, like oatmeal, melon and baked squash, were typically among my favorites. Bratwurst (the worst and potentially most lethal kind of food in my perception) was rated an A+! If I wanted to avoid glucose spikes, I’d have to make some pretty big sacrifices in my diet.

Nevertheless, it was an interesting first step on the path to a personalized diet. There is now a commercial version of this test, based on the research of Dr. Segal and Dr. Elinav, though it is much more limited: It only analyzes a gut microbiome sample, without monitoring glucose or what you eat.

There are other efforts underway in the field as well. In some continuing nutrition studies, smartphone photos of participants’ plates of food are being processed by deep learning, another subtype of A.I., to accurately determine what they are eating. This avoids the hassle of manually logging in the data and the use of unreliable food diaries (as long as participants remember to take the picture).

But that is a single type of data. What we really need to do is pull in multiple types of data — activity, sleep, level of stress, medications, genome, microbiome and glucose — from multiple devices, like skin patches and smartwatches. With advanced algorithms, this is eminently doable. In the next few years, you could have a virtual health coach that is deep learning about your relevant health metrics and providing you with customized dietary recommendations.

The benefits of such a coach will, of course, have to be validated by randomized trials, unlike the myriad diets that are being hawked without any proof that they are effective or even safe.

We don’t often think of a diet as being unsafe, but the wrong foods can be dangerous for people with certain risks or conditions. I’ve had two bouts of kidney stones. To avoid a third, I need to stay away from foods high in oxalate, a naturally occurring molecule abundant in plants. But if you look at the recommendations for my personalized diet, many — like nuts and strawberries — are high in oxalate. That’s a big miscue, because my pre-existing medical conditions were not one of the test’s inputs. And as we undergo significant changes through our lives, like pregnancy or aging, we’ll need re-assessments of what our optimal diet should be.

For now, it’s striking that it took big data and A.I. to reboot our perceptions about something as fundamental as what we eat. We’re still a ways away from “You Paleo, me Keto,” but at least we’re finally making progress, learning that there is no such thing as a universal diet.

Eric Topol (@EricTopol), a cardiologist and professor of molecular medicine, is the executive vice president of Scripps Research. He is the author of the forthcoming “Deep Medicine,” from which this essay is adapted.

The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: letters@nytimes.com.

Follow The New York Times Opinion section on Facebook, Twitter (@NYTopinion) and Instagram.

from Hacker News https://ift.tt/YV9WJO
via IFTTT

Deep learning may need a new programming language

http://bit.ly/2TVQezC

Deep learning may need a new programming language that’s more flexible and easier to work with than Python, Facebook AI Research director Yann LeCun said today. It’s not yet clear if such a language is necessary, but the possibility runs against very entrenched desires from researchers and engineers, he said.

LeCun has worked with neural networks since the 1980s.

“There are several projects at Google, Facebook, and other places to kind of design such a compiled language that can be efficient for deep learning, but it’s not clear at all that the community will follow, because people just want to use Python,” LeCun said in a phone call with VentureBeat.

“The question now is, is that a valid approach?”

Python is currently the most popular language used by developers working on machine learning projects, according to GitHub’s recent Octoverse report, and the language forms the basis for Facebook’s PyTorch and Google’s TensorFlow frameworks.

LeCun presented a paper exploring the latest trends and spoke before companies making next generation computer chips at the IEEE’s International Solid-State Circuits Conference (ISSCC) today in San Francisco.

The first portion of the paper is devoted to lessons LeCun took away from Bell Labs, including his observation that the AI researchers’ and computer scientists’ imaginations tend to be tied to hardware and software tools.

Artificial intelligence is more than 50 years old, but its current rise has been closely linked to the growth in compute power provided by computer chips and other hardware.

A virtuous cycle of better hardware causing better algorithms, causing better performance, causing more people to build better hardware is only a few years old, said LeCun, who worked at Bell Labs in the 1980s and made ConvNet (CNN) AI to read zip codes on postal envelopes and bank checks.

In the early 2000s, after leaving Bell Labs and joining New York University, LeCun worked with other luminaries in the space, like Yoshua Bengio and Geoffrey Hinton, conducting research to revive interest in neural networks and grow the popularity of deep learning.

In recent years, advances in hardware — like field programmable gate arrays (FPGA), tensor processing units (TPU) from Google, and graphics processing units (GPU) — have played a major role in the industry’s growth. Facebook is reportedly also working on its own semiconductor.

“The kind of hardware that’s available has a big influence on the kind of research that people do, and so the direction of AI in the next decade or so is going to be greatly influenced by what hardware becomes available,” he said. “It’s very humbling for computer scientists, because we like to think in the abstract that we’re not bound by the limitation of our hardware, but in fact we are.”

LeCun highlighted a number of AI trends hardware makers should consider in the years ahead and made recommendations about the kind of architecture needed in the near future, recommending that the growing size of deep learning systems be taken into consideration.

He also spoke about the need for hardware designed specifically for deep learning and hardware that can handle a batch of one, rather than needing to batch multiple training samples to efficiently run a neural net, which is the standard today.

“If you run a single image, you’re not going to be able to exploit all the computation that’s available to you in a GPU. You’re going to waste resources, basically, so batching forces you to think about certain ways of training neural nets,” he said.

He also recommended dynamic networks and hardware that can adjust to utilize only the neurons needed for a task.

In the paper, LeCun reiterated his belief that self-supervised learning will play a major role in advancing state-of-the art AI.

“If self-supervised learning eventually allows machines to learn vast amounts of background knowledge about how the world works through observation, one may hypothesize that some form of machine common sense could emerge,” LeCun wrote in the paper.

LeCun believes that future deep learning systems will largely be trained with self-supervised learning and that new high-performance hardware will be needed to support such self-supervised learning.

Last month, LeCun discussed the importance of self-supervised learning with VentureBeat as part of a story about predictions for AI in 2019. Hardware that can handle self-supervised learning will be important for Facebook, as well as for autonomous driving, robotics, and many other forms of technology.

from Hacker News http://bit.ly/YV9WJO
via IFTTT

A Plan for Spam (2002)

http://bit.ly/qPHWag

from Hacker News http://bit.ly/YV9WJO
via IFTTT

August 2002

(This article describes the spam-filtering techniques used in the spamproof web-based mail reader we built to exercise Arc. An improved algorithm is described in Better Bayesian Filtering.)

I think it's possible to stop spam, and that content-based filters are the way to do it. The Achilles heel of the spammers is their message. They can circumvent any other barrier you set up. They have so far, at least. But they have to deliver their message, whatever it is. If we can write software that recognizes their messages, there is no way they can get around that.

_ _ _

To the recipient, spam is easily recognizable. If you hired someone to read your mail and discard the spam, they would have little trouble doing it. How much do we have to do, short of AI, to automate this process?

I think we will be able to solve the problem with fairly simple algorithms. In fact, I've found that you can filter present-day spam acceptably well using nothing more than a Bayesian combination of the spam probabilities of individual words. Using a slightly tweaked (as described below) Bayesian filter, we now miss less than 5 per 1000 spams, with 0 false positives.

The statistical approach is not usually the first one people try when they write spam filters. Most hackers' first instinct is to try to write software that recognizes individual properties of spam. You look at spams and you think, the gall of these guys to try sending me mail that begins "Dear Friend" or has a subject line that's all uppercase and ends in eight exclamation points. I can filter out that stuff with about one line of code.

And so you do, and in the beginning it works. A few simple rules will take a big bite out of your incoming spam. Merely looking for the word "click" will catch 79.7% of the emails in my spam corpus, with only 1.2% false positives.

I spent about six months writing software that looked for individual spam features before I tried the statistical approach. What I found was that recognizing that last few percent of spams got very hard, and that as I made the filters stricter I got more false positives.

False positives are innocent emails that get mistakenly identified as spams. For most users, missing legitimate email is an order of magnitude worse than receiving spam, so a filter that yields false positives is like an acne cure that carries a risk of death to the patient.

The more spam a user gets, the less likely he'll be to notice one innocent mail sitting in his spam folder. And strangely enough, the better your spam filters get, the more dangerous false positives become, because when the filters are really good, users will be more likely to ignore everything they catch.

I don't know why I avoided trying the statistical approach for so long. I think it was because I got addicted to trying to identify spam features myself, as if I were playing some kind of competitive game with the spammers. (Nonhackers don't often realize this, but most hackers are very competitive.) When I did try statistical analysis, I found immediately that it was much cleverer than I had been. It discovered, of course, that terms like "virtumundo" and "teens" were good indicators of spam. But it also discovered that "per" and "FL" and "ff0000" are good indicators of spam. In fact, "ff0000" (html for bright red) turns out to be as good an indicator of spam as any pornographic term.

_ _ _

Here's a sketch of how I do statistical filtering. I start with one corpus of spam and one of nonspam mail. At the moment each one has about 4000 messages in it. I scan the entire text, including headers and embedded html and javascript, of each message in each corpus. I currently consider alphanumeric characters, dashes, apostrophes, and dollar signs to be part of tokens, and everything else to be a token separator. (There is probably room for improvement here.) I ignore tokens that are all digits, and I also ignore html comments, not even considering them as token separators.

I count the number of times each token (ignoring case, currently) occurs in each corpus. At this stage I end up with two large hash tables, one for each corpus, mapping tokens to number of occurrences.

Next I create a third hash table, this time mapping each token to the probability that an email containing it is a spam, which I calculate as follows [1]: (let ((g (* 2 (or (gethash word good) 0))) (b (or (gethash word bad) 0))) (unless (< (+ g b) 5) (max .01 (min .99 (float (/ (min 1 (/ b nbad)) (+ (min 1 (/ g ngood)) (min 1 (/ b nbad))))))))) where word is the token whose probability we're calculating, good and bad are the hash tables I created in the first step, and ngood and nbad are the number of nonspam and spam messages respectively.

I explained this as code to show a couple of important details. I want to bias the probabilities slightly to avoid false positives, and by trial and error I've found that a good way to do it is to double all the numbers in good. This helps to distinguish between words that occasionally do occur in legitimate email and words that almost never do. I only consider words that occur more than five times in total (actually, because of the doubling, occurring three times in nonspam mail would be enough). And then there is the question of what probability to assign to words that occur in one corpus but not the other. Again by trial and error I chose .01 and .99. There may be room for tuning here, but as the corpus grows such tuning will happen automatically anyway.

The especially observant will notice that while I consider each corpus to be a single long stream of text for purposes of counting occurrences, I use the number of emails in each, rather than their combined length, as the divisor in calculating spam probabilities. This adds another slight bias to protect against false positives.

When new mail arrives, it is scanned into tokens, and the most interesting fifteen tokens, where interesting is measured by how far their spam probability is from a neutral .5, are used to calculate the probability that the mail is spam. If probs is a list of the fifteen individual probabilities, you calculate the combined probability thus: (let ((prod (apply #'* probs))) (/ prod (+ prod (apply #'* (mapcar #'(lambda (x) (- 1 x)) probs))))) One question that arises in practice is what probability to assign to a word you've never seen, i.e. one that doesn't occur in the hash table of word probabilities. I've found, again by trial and error, that .4 is a good number to use. If you've never seen a word before, it is probably fairly innocent; spam words tend to be all too familiar.

There are examples of this algorithm being applied to actual emails in an appendix at the end.

I treat mail as spam if the algorithm above gives it a probability of more than .9 of being spam. But in practice it would not matter much where I put this threshold, because few probabilities end up in the middle of the range.

_ _ _

One great advantage of the statistical approach is that you don't have to read so many spams. Over the past six months, I've read literally thousands of spams, and it is really kind of demoralizing. Norbert Wiener said if you compete with slaves you become a slave, and there is something similarly degrading about competing with spammers. To recognize individual spam features you have to try to get into the mind of the spammer, and frankly I want to spend as little time inside the minds of spammers as possible.

But the real advantage of the Bayesian approach, of course, is that you know what you're measuring. Feature-recognizing filters like SpamAssassin assign a spam "score" to email. The Bayesian approach assigns an actual probability. The problem with a "score" is that no one knows what it means. The user doesn't know what it means, but worse still, neither does the developer of the filter. How many points should an email get for having the word "sex" in it? A probability can of course be mistaken, but there is little ambiguity about what it means, or how evidence should be combined to calculate it. Based on my corpus, "sex" indicates a .97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability. And Bayes' Rule, equally unambiguous, says that an email containing both words would, in the (unlikely) absence of any other evidence, have a 99.97% chance of being a spam.

Because it is measuring probabilities, the Bayesian approach considers all the evidence in the email, both good and bad. Words that occur disproportionately rarely in spam (like "though" or "tonight" or "apparently") contribute as much to decreasing the probability as bad words like "unsubscribe" and "opt-in" do to increasing it. So an otherwise innocent email that happens to include the word "sex" is not going to get tagged as spam.

Ideally, of course, the probabilities should be calculated individually for each user. I get a lot of email containing the word "Lisp", and (so far) no spam that does. So a word like that is effectively a kind of password for sending mail to me. In my earlier spam-filtering software, the user could set up a list of such words and mail containing them would automatically get past the filters. On my list I put words like "Lisp" and also my zipcode, so that (otherwise rather spammy-sounding) receipts from online orders would get through. I thought I was being very clever, but I found that the Bayesian filter did the same thing for me, and moreover discovered of a lot of words I hadn't thought of.

When I said at the start that our filters let through less than 5 spams per 1000 with 0 false positives, I'm talking about filtering my mail based on a corpus of my mail. But these numbers are not misleading, because that is the approach I'm advocating: filter each user's mail based on the spam and nonspam mail he receives. Essentially, each user should have two delete buttons, ordinary delete and delete-as-spam. Anything deleted as spam goes into the spam corpus, and everything else goes into the nonspam corpus.

You could start users with a seed filter, but ultimately each user should have his own per-word probabilities based on the actual mail he receives. This (a) makes the filters more effective, (b) lets each user decide their own precise definition of spam, and (c) perhaps best of all makes it hard for spammers to tune mails to get through the filters. If a lot of the brain of the filter is in the individual databases, then merely tuning spams to get through the seed filters won't guarantee anything about how well they'll get through individual users' varying and much more trained filters.

Content-based spam filtering is often combined with a whitelist, a list of senders whose mail can be accepted with no filtering. One easy way to build such a whitelist is to keep a list of every address the user has ever sent mail to. If a mail reader has a delete-as-spam button then you could also add the from address of every email the user has deleted as ordinary trash.

I'm an advocate of whitelists, but more as a way to save computation than as a way to improve filtering. I used to think that whitelists would make filtering easier, because you'd only have to filter email from people you'd never heard from, and someone sending you mail for the first time is constrained by convention in what they can say to you. Someone you already know might send you an email talking about sex, but someone sending you mail for the first time would not be likely to. The problem is, people can have more than one email address, so a new from-address doesn't guarantee that the sender is writing to you for the first time. It is not unusual for an old friend (especially if he is a hacker) to suddenly send you an email with a new from-address, so you can't risk false positives by filtering mail from unknown addresses especially stringently.

In a sense, though, my filters do themselves embody a kind of whitelist (and blacklist) because they are based on entire messages, including the headers. So to that extent they "know" the email addresses of trusted senders and even the routes by which mail gets from them to me. And they know the same about spam, including the server names, mailer versions, and protocols.

_ _ _

If I thought that I could keep up current rates of spam filtering, I would consider this problem solved. But it doesn't mean much to be able to filter out most present-day spam, because spam evolves. Indeed, most antispam techniques so far have been like pesticides that do nothing more than create a new, resistant strain of bugs.

I'm more hopeful about Bayesian filters, because they evolve with the spam. So as spammers start using "c0ck" instead of "cock" to evade simple-minded spam filters based on individual words, Bayesian filters automatically notice. Indeed, "c0ck" is far more damning evidence than "cock", and Bayesian filters know precisely how much more.

Still, anyone who proposes a plan for spam filtering has to be able to answer the question: if the spammers knew exactly what you were doing, how well could they get past you? For example, I think that if checksum-based spam filtering becomes a serious obstacle, the spammers will just switch to mad-lib techniques for generating message bodies.

To beat Bayesian filters, it would not be enough for spammers to make their emails unique or to stop using individual naughty words. They'd have to make their mails indistinguishable from your ordinary mail. And this I think would severely constrain them. Spam is mostly sales pitches, so unless your regular mail is all sales pitches, spams will inevitably have a different character. And the spammers would also, of course, have to change (and keep changing) their whole infrastructure, because otherwise the headers would look as bad to the Bayesian filters as ever, no matter what they did to the message body. I don't know enough about the infrastructure that spammers use to know how hard it would be to make the headers look innocent, but my guess is that it would be even harder than making the message look innocent.

Assuming they could solve the problem of the headers, the spam of the future will probably look something like this: Hey there. Thought you should check out the following: http://bit.ly/2V4Cq63 because that is about as much sales pitch as content-based filtering will leave the spammer room to make. (Indeed, it will be hard even to get this past filters, because if everything else in the email is neutral, the spam probability will hinge on the url, and it will take some effort to make that look neutral.)

Spammers range from businesses running so-called opt-in lists who don't even try to conceal their identities, to guys who hijack mail servers to send out spams promoting porn sites. If we use filtering to whittle their options down to mails like the one above, that should pretty much put the spammers on the "legitimate" end of the spectrum out of business; they feel obliged by various state laws to include boilerplate about why their spam is not spam, and how to cancel your "subscription," and that kind of text is easy to recognize.

(I used to think it was naive to believe that stricter laws would decrease spam. Now I think that while stricter laws may not decrease the amount of spam that spammers send, they can certainly help filters to decrease the amount of spam that recipients actually see.)

All along the spectrum, if you restrict the sales pitches spammers can make, you will inevitably tend to put them out of business. That word business is an important one to remember. The spammers are businessmen. They send spam because it works. It works because although the response rate is abominably low (at best 15 per million, vs 3000 per million for a catalog mailing), the cost, to them, is practically nothing. The cost is enormous for the recipients, about 5 man-weeks for each million recipients who spend a second to delete the spam, but the spammer doesn't have to pay that.

Sending spam does cost the spammer something, though. [2] So the lower we can get the response rate-- whether by filtering, or by using filters to force spammers to dilute their pitches-- the fewer businesses will find it worth their while to send spam.

The reason the spammers use the kinds of sales pitches that they do is to increase response rates. This is possibly even more disgusting than getting inside the mind of a spammer, but let's take a quick look inside the mind of someone who responds to a spam. This person is either astonishingly credulous or deeply in denial about their sexual interests. In either case, repulsive or idiotic as the spam seems to us, it is exciting to them. The spammers wouldn't say these things if they didn't sound exciting. And "thought you should check out the following" is just not going to have nearly the pull with the spam recipient as the kinds of things that spammers say now. Result: if it can't contain exciting sales pitches, spam becomes less effective as a marketing vehicle, and fewer businesses want to use it.

That is the big win in the end. I started writing spam filtering software because I didn't want have to look at the stuff anymore. But if we get good enough at filtering out spam, it will stop working, and the spammers will actually stop sending it.

_ _ _

Of all the approaches to fighting spam, from software to laws, I believe Bayesian filtering will be the single most effective. But I also think that the more different kinds of antispam efforts we undertake, the better, because any measure that constrains spammers will tend to make filtering easier. And even within the world of content-based filtering, I think it will be a good thing if there are many different kinds of software being used simultaneously. The more different filters there are, the harder it will be for spammers to tune spams to get through them.

Appendix: Examples of Filtering

Here is an example of a spam that arrived while I was writing this article. The fifteen most interesting words in this spam are: qvp0045 indira mx-05 intimail $7500 freeyankeedom cdo bluefoxmedia jpg unsecured platinum 3d0 qves 7c5 7c266675 The words are a mix of stuff from the headers and from the message body, which is typical of spam. Also typical of spam is that every one of these words has a spam probability, in my database, of .99. In fact there are more than fifteen words with probabilities of .99, and these are just the first fifteen seen.

Unfortunately that makes this email a boring example of the use of Bayes' Rule. To see an interesting variety of probabilities we have to look at this actually quite atypical spam.

The fifteen most interesting words in this spam, with their probabilities, are: madam 0.99 promotion 0.99 republic 0.99 shortest 0.047225013 mandatory 0.047225013 standardization 0.07347802 sorry 0.08221981 supported 0.09019077 people's 0.09019077 enter 0.9075001 quality 0.8921298 organization 0.12454646 investment 0.8568143 very 0.14758544 valuable 0.82347786 This time the evidence is a mix of good and bad. A word like "shortest" is almost as much evidence for innocence as a word like "madam" or "promotion" is for guilt. But still the case for guilt is stronger. If you combine these numbers according to Bayes' Rule, the resulting probability is .9027.

"Madam" is obviously from spams beginning "Dear Sir or Madam." They're not very common, but the word "madam" never occurs in my legitimate email, and it's all about the ratio.

"Republic" scores high because it often shows up in Nigerian scam emails, and also occurs once or twice in spams referring to Korea and South Africa. You might say that it's an accident that it thus helps identify this spam. But I've found when examining spam probabilities that there are a lot of these accidents, and they have an uncanny tendency to push things in the right direction rather than the wrong one. In this case, it is not entirely a coincidence that the word "Republic" occurs in Nigerian scam emails and this spam. There is a whole class of dubious business propositions involving less developed countries, and these in turn are more likely to have names that specify explicitly (because they aren't) that they are republics.[3]

On the other hand, "enter" is a genuine miss. It occurs mostly in unsubscribe instructions, but here is used in a completely innocent way. Fortunately the statistical approach is fairly robust, and can tolerate quite a lot of misses before the results start to be thrown off.

For comparison, here is an example of that rare bird, a spam that gets through the filters. Why? Because by sheer chance it happens to be loaded with words that occur in my actual email: perl 0.01 python 0.01 tcl 0.01 scripting 0.01 morris 0.01 graham 0.01491078 guarantee 0.9762507 cgi 0.9734398 paul 0.027040077 quite 0.030676773 pop3 0.042199217 various 0.06080265 prices 0.9359873 managed 0.06451222 difficult 0.071706355 There are a couple pieces of good news here. First, this mail probably wouldn't get through the filters of someone who didn't happen to specialize in programming languages and have a good friend called Morris. For the average user, all the top five words here would be neutral and would not contribute to the spam probability.

Second, I think filtering based on word pairs (see below) might well catch this one: "cost effective", "setup fee", "money back" -- pretty incriminating stuff. And of course if they continued to spam me (or a network I was part of), "Hostex" itself would be recognized as a spam term.

Finally, here is an innocent email. Its fifteen most interesting words are as follows: continuation 0.01 describe 0.01 continuations 0.01 example 0.033600237 programming 0.05214485 i'm 0.055427782 examples 0.07972858 color 0.9189189 localhost 0.09883721 hi 0.116539136 california 0.84421706 same 0.15981844 spot 0.1654587 us-ascii 0.16804294 what 0.19212411 Most of the words here indicate the mail is an innocent one. There are two bad smelling words, "color" (spammers love colored fonts) and "California" (which occurs in testimonials and also in menus in forms), but they are not enough to outweigh obviously innocent words like "continuation" and "example".

It's interesting that "describe" rates as so thoroughly innocent. It hasn't occurred in a single one of my 4000 spams. The data turns out to be full of such surprises. One of the things you learn when you analyze spam texts is how narrow a subset of the language spammers operate in. It's that fact, together with the equally characteristic vocabulary of any individual user's mail, that makes Bayesian filtering a good bet.

Appendix: More Ideas

One idea that I haven't tried yet is to filter based on word pairs, or even triples, rather than individual words. This should yield a much sharper estimate of the probability. For example, in my current database, the word "offers" has a probability of .96. If you based the probabilities on word pairs, you'd end up with "special offers" and "valuable offers" having probabilities of .99 and, say, "approach offers" (as in "this approach offers") having a probability of .1 or less.

The reason I haven't done this is that filtering based on individual words already works so well. But it does mean that there is room to tighten the filters if spam gets harder to detect. (Curiously, a filter based on word pairs would be in effect a Markov-chaining text generator running in reverse.)

Specific spam features (e.g. not seeing the recipient's address in the to: field) do of course have value in recognizing spam. They can be considered in this algorithm by treating them as virtual words. I'll probably do this in future versions, at least for a handful of the most egregious spam indicators. Feature-recognizing spam filters are right in many details; what they lack is an overall discipline for combining evidence.

Recognizing nonspam features may be more important than recognizing spam features. False positives are such a worry that they demand extraordinary measures. I will probably in future versions add a second level of testing designed specifically to avoid false positives. If a mail triggers this second level of filters it will be accepted even if its spam probability is above the threshold.

I don't expect this second level of filtering to be Bayesian. It will inevitably be not only ad hoc, but based on guesses, because the number of false positives will not tend to be large enough to notice patterns. (It is just as well, anyway, if a backup system doesn't rely on the same technology as the primary system.)

Another thing I may try in the future is to focus extra attention on specific parts of the email. For example, about 95% of current spam includes the url of a site they want you to visit. (The remaining 5% want you to call a phone number, reply by email or to a US mail address, or in a few cases to buy a certain stock.) The url is in such cases practically enough by itself to determine whether the email is spam.

Domain names differ from the rest of the text in a (non-German) email in that they often consist of several words stuck together. Though computationally expensive in the general case, it might be worth trying to decompose them. If a filter has never seen the token "xxxporn" before it will have an individual spam probability of .4, whereas "xxx" and "porn" individually have probabilities (in my corpus) of .9889 and .99 respectively, and a combined probability of .9998.

I expect decomposing domain names to become more important as spammers are gradually forced to stop using incriminating words in the text of their messages. (A url with an ip address is of course an extremely incriminating sign, except in the mail of a few sysadmins.)

It might be a good idea to have a cooperatively maintained list of urls promoted by spammers. We'd need a trust metric of the type studied by Raph Levien to prevent malicious or incompetent submissions, but if we had such a thing it would provide a boost to any filtering software. It would also be a convenient basis for boycotts.

Another way to test dubious urls would be to send out a crawler to look at the site before the user looked at the email mentioning it. You could use a Bayesian filter to rate the site just as you would an email, and whatever was found on the site could be included in calculating the probability of the email being a spam. A url that led to a redirect would of course be especially suspicious.

One cooperative project that I think really would be a good idea would be to accumulate a giant corpus of spam. A large, clean corpus is the key to making Bayesian filtering work well. Bayesian filters could actually use the corpus as input. But such a corpus would be useful for other kinds of filters too, because it could be used to test them.

Creating such a corpus poses some technical problems. We'd need trust metrics to prevent malicious or incompetent submissions, of course. We'd also need ways of erasing personal information (not just to-addresses and ccs, but also e.g. the arguments to unsubscribe urls, which often encode the to-address) from mails in the corpus. If anyone wants to take on this project, it would be a good thing for the world.

Appendix: Defining Spam

I think there is a rough consensus on what spam is, but it would be useful to have an explicit definition. We'll need to do this if we want to establish a central corpus of spam, or even to compare spam filtering rates meaningfully.

To start with, spam is not unsolicited commercial email. If someone in my neighborhood heard that I was looking for an old Raleigh three-speed in good condition, and sent me an email offering to sell me one, I'd be delighted, and yet this email would be both commercial and unsolicited. The defining feature of spam (in fact, its raison d'etre) is not that it is unsolicited, but that it is automated.

It is merely incidental, too, that spam is usually commercial. If someone started sending mass email to support some political cause, for example, it would be just as much spam as email promoting a porn site.

I propose we define spam as unsolicited automated email. This definition thus includes some email that many legal definitions of spam don't. Legal definitions of spam, influenced presumably by lobbyists, tend to exclude mail sent by companies that have an "existing relationship" with the recipient. But buying something from a company, for example, does not imply that you have solicited ongoing email from them. If I order something from an online store, and they then send me a stream of spam, it's still spam.

Companies sending spam often give you a way to "unsubscribe," or ask you to go to their site and change your "account preferences" if you want to stop getting spam. This is not enough to stop the mail from being spam. Not opting out is not the same as opting in. Unless the recipient explicitly checked a clearly labelled box (whose default was no) asking to receive the email, then it is spam.

In some business relationships, you do implicitly solicit certain kinds of mail. When you order online, I think you implicitly solicit a receipt, and notification when the order ships. I don't mind when Verisign sends me mail warning that a domain name is about to expire (at least, if they are the actual registrar for it). But when Verisign sends me email offering a FREE Guide to Building My E-Commerce Web Site, that's spam.

Notes:

[1] The examples in this article are translated into Common Lisp for, believe it or not, greater accessibility. The application described here is one that we wrote in order to test a new Lisp dialect called Arc that is not yet released.

[2] Currently the lowest rate seems to be about $200 to send a million spams. That's very cheap, 1/50th of a cent per spam. But filtering out 95% of spam, for example, would increase the spammers' cost to reach a given audience by a factor of 20. Few can have margins big enough to absorb that.

[3] As a rule of thumb, the more qualifiers there are before the name of a country, the more corrupt the rulers. A country called The Socialist People's Democratic Republic of X is probably the last place in the world you'd want to live.

Thanks to Sarah Harlin for reading drafts of this; Daniel Giffin (who is also writing the production Arc interpreter) for several good ideas about filtering and for creating our mail infrastructure; Robert Morris, Trevor Blackwell and Erann Gat for many discussions about spam; Raph Levien for advice about trust metrics; and Chip Coldwell and Sam Steingold for advice about statistics.

More Info:

Oculus Studios and Sanzaru unveil Asgard’s Wrath VR game

http://bit.ly/2RFgtZh

Facebook’s Oculus Studios division announced a new virtual reality game Asgard’s Wrath, which is coming for the Oculus Rift virtual reality headset from Sanzaru Games.

The title shows that Oculus remains committed to publishing games on its VR platform in hopes of distinguishing it from the competition in the growing VR market.

Mike Doran, an executive producer at Oculus Studios, in a blog post that Asgard’s Wrath that players will battle as a mortal and rise as a god in this Norse-inspired action adventure with a heavy role-playing game (RPG) elements. It‘s coming to Rift in 2019.

Above: Asgard’s Wrath comes out in 2019 on the Oculus Rift.

Image Credit: Oculus

In the story, the gods are in their twilight, with Asgard’s inhabitants consumed by bickering and selfish exploits. You, Fledgling God, are birthed in an explosion of light—a clash of primordial forces of nature. Your story begins in the middle of the action with a dramatic, action-packed encounter with Loki.

He takes great interest in your potential. Before he can make you a god, Loki has several requests to test your worthiness. Each revolves around a standalone scenario, or Saga, where you must use your powers to help preordained Heroes of the Realms fulfill their destinies.

Sanzaru has been working on this title for years. With four movement-heavy VR titles under its belt, Sanzaru is exploring physicality and translating full-body movement into an immersive VR experience with realistic physics.

Above: Asgard’s Wrath features realistic physic in combat.

Image Credit: Oculus

“I’ve had the honor of collaborating with them on Asgard’s Wrath through many different experiments and genre shifts that got us to what you see today, and that’s only a brief glimpse of what’s in store,” Doran said.

He said, “Sanzaru has cracked the code on bringing visceral melee combat to VR. The fighting mechanics in Asgard’s Wrath anchor the title, require skill, and encourage physical movement. It’s visceral, satisfying, and just fun.”

Asgard’s Wrath’s melee fighting system leverages physics and is supported by game rules that demand—and reward—playing with skill. In this game, you can’t just swing wildly and be successful. That might work against some lesser enemies, but eventually you’re going to get your butt handed to you that way, Doran said.

“We’ve also incorporated some brutally satisfying dismemberment,” he said. “One of my favorite things to do in-game is to use a shield to ‘catch’ thrown enemy weapons like daggers, yank them out, and throw them right back. Another great moment is swinging your weapon to knock decapitated monster heads through the air.”

The game brings together physics, time dilation, dismemberment, and ragdoll body movements to complete the core foundation of combat.

When Sanzaru first started prototyping this game, the player was always at god-scale. Eventually, the developers added a mortal hero to mix things up and give the player something to protect and support. Sanzaru then had the idea to let players swap back and forth between god and hero.

from VentureBeat https://venturebeat.com
via IFTTT

Denver to vote on whether to decriminalize 'magic mushrooms'

https://reut.rs/2G8P68N

DENVER (Reuters) - Denver voters will decide in May whether to decriminalize possession of small amounts of the hallucinogenic drug psilocybin, which would make it the first U.S. city to halt prosecution of people caught with psychedelic mushrooms.

The citizen-driven proposal, which election officials said this week reached the required number of signatures to be on the city’s municipal ballot, would not legalize so-called “magic mushrooms,” but rather make them a low priority for law enforcement, according to its language.

Decriminalize Denver, the group behind the ballot question, said the drug has medical benefits that could reduce psychological stress and opioid dependence.

“Nationally, Denver and the state of Colorado have represented the first movers in a revised understanding of the potential benefits of naturally-occurring psychoactive medicines,” the group said on its website.

Some opponents worry that if passed the ordinance would further tarnish the city’s image, given that recreational marijuana is already allowed under Colorado law, and another proposal by the city to create the country’s first safe injection site for intravenous drug users was approved by the city council in November.

“Denver is quickly becoming the illicit drug capitol of the world,” Jeff Hunt, director of the Colorado-based Centennial Institute, a conservative think tank, said in a statement. “High potency pot, proposed needle injection sites, and now an effort to decriminalize mushrooms.”

The safe injection site pilot program would need the approval of the state legislature, which has not yet taken up the issue. Federal authorities have warned that such a facility would be illegal.

Kevin Matthews, 33, campaign director for Decriminalize Denver, said worries about expanded drug use under the measure are unwarranted.

“Nothing on our ballot question would do anything to increase access – it does not allow for distribution and sale,” Matthews told Reuters in a phone interview, adding that mushrooms have helped treat his depression.

Mayor Michael Hancock told the Denver Post that he opposes the mushroom question.

Psilocybin is illegal under both Colorado and federal law. The U.S. Drug Enforcement Administration classifies the drug as a Schedule 1 substance, meaning the agency has deemed that it has a high potential for abuse and currently has no accepted medical use.

In 2004, Denver voters voted to decriminalize marijuana possession, years before Colorado voters voted to approve its legalization for recreational use and establish a full regulatory framework.

Reporting by Keith Coffman; Editing by Daniel Wallis

from Hacker News http://bit.ly/YV9WJO
via IFTTT

How to do a digital detox... - Bdaily

http://bit.ly/2S0MqjQ

How to do a digital detox... Bdaily

As the the number of online devices we own continues to grow, more and more of us are trying to allow ourselves to be more “in the moment” by consciously ...

from "Simon Sinek" - Google News http://bit.ly/2Uz3KZT
via IFTTT

I Am Mother is a slow, tense movie about how we love and fear AI

http://bit.ly/2Tg5CXn

Welcome to Cheat Sheet, our brief breakdown-style reviews of festival films, VR previews, and other special event releases. This review comes from the 2019 Sundance Film Festival.

When a fictional AI “goes rogue,” that often really means that it’s working exactly as intended. Tell a machine to make paperclips, and it will turn the entire world into little twists of metal. Ask it to save the planet, and it will decide that people are Earth’s greatest threat. We dream of creating machines that are smarter, more ethical, and more logical than ourselves. Then we fear where that logic will take them.

I Am Mother, the debut film from Australian director Grant Sputore, follows in this tradition. It’s a story about the bond between a mother and child where the mother is a robot, the child was an artificially gestated embryo and is known only as Daughter, and they’re both living in an airtight shelter after the end of the world. I Am Mother doesn’t plumb the potential weirdness of that premise, and it’s working in a well-worn genre without breaking much new ground. But it effectively dramatizes our perennial love-hate relationship with artificial intelligence.

This review contains spoilers for early sections of I Am Mother.

Photo by Ian Routledge / Courtesy of Sundance Institute

What’s the genre?

Highbrow science fiction thriller. I Am Mother’s first act is slow and meditative, set entirely within a sterile, spaceship-like bunker. It offers little backstory, although the design of the parental AI Mother — based on real robots like Boston Dynamics’ Atlas — grounds the world in near-future territory. The film mostly focuses on its characters’ mundane but eerie daily routines, occasionally throwing in quirky details about their lives, like Daughter’s love of Johnny Carson-era Tonight Show episodes.

After I Am Mother’s first major twist, though, the film becomes less enigmatic and more plot-driven, as Daughter (Clara Rugaard) tries to figure out Mother’s real agenda. By the end, it’s a fairly traditional science fiction action movie.

What’s it about?

Years after a mysterious “extinction event,” a hulking but soft-voiced robot known as Mother (voiced by Rose Byrne) is raising the first of a new generation of humans. Daughter has been trained in advanced engineering and medical skills as well as the intricacies of moral philosophy. She believes the outside world is lifeless and ravaged by disease, thanks to humans’ self-destructive behavior — until an injured woman (Hilary Swank) shows up at the bunker’s airlock, begging for help.

The survivor’s presence proves that at least some of Mother’s stories are lies, but the woman is cagey about the truth, which sets up a conflict where Daughter has no idea what’s true or who to trust. When information starts emerging, Mother insists that the woman can’t be trusted and that she has her own selfish reasons for gaining Daughter’s confidence. Daughter has to navigate the conflicting stories and her own dreams of building a bigger family in a human world. Then she has to decide whether to stay with Mother in the bunker or help the woman escape, against Mother’s express wishes.

What’s it really about?

How robots are probably going to destroy us, and we’ll have only ourselves to blame. After the premiere, Sputore described I Am Mother as “largely a study about what it means to be good.” Its plot hinges on how artificial intelligence might interpret goodness in a world where people seem hellbent on driving themselves toward extinction, and “robots will either save us from that, or they’ll probably expedite it.” In I Am Mother, Swank’s character chips away at Daughter’s unquestioning faith in Mother, who is more powerful and dangerous than she initially appears. But the woman also justifies Mother’s concerns about humanity.

On a more personal level, I Am Mother is about a child learning that parental figures aren’t objective, infallible beings. Its best moments follow Daughter as she navigates the power struggle between two potential mothers: one offering safety and self-actualization, the other promising freedom and companionship, and both telling half-truths about their motivations.

Is it good?

I Am Mother’s premise may seem awfully familiar to anybody who’s watched a few killer-robot movies. The more secrets are revealed and ethical conundrums are posed, the less compelling the story becomes because every twist makes it more reminiscent of other high-concept dystopian films. That’s especially true with Mother, who is most interesting in her first iteration where she’s hyper-competent but still vulnerable and still in the process of learning about parenthood.

The film loses track of its characters’ relationships when it focuses too much on utilitarian thought experiments. It also wastes the story’s strongest conceit: a protagonist who seems deeply ambivalent about humanity and how she deals with meeting a real person for the first time. One of I Am Mother’s early scenes has Daughter lamenting the fact that she’s human because “they ruined everything.” Beyond watching celebrities on late-night talk shows, her experience with human behavior seems limited to discussing Kant and trolley problems with Mother.

Initially, the injured woman is furious at Daughter for forcing her to trust a machine, and their mutual antagonism might have made for an interesting relationship, with Swank’s grim, violent, manipulative character representing everything wrong with the old world. Instead, Daughter responds to her like any standard post-apocalyptic vault-dweller, blandly curious about life outside the bunker.

But while parts of I Am Mother are frustratingly generic, Mother’s physical design — conceived by New Zealand special effects studio Weta Workshop, and played by effects artist Luke Hawker in a robot suit — is distinctively imposing. I Am Mother breaks with the convention of depicting powerful AI as bodiless and omnipresent, opting to obscure exactly how much Mother hears and sees. The robot’s blocky figure, which a young Daughter adorns with colorful stickers, contrasts charmingly with her gentle mannerisms.

As the pair’s relationship deteriorates, though, watching a worried Mother charge down corridors inspires the same uneasiness as those Boston Dynamics videos of robots doing parkour or running obstacle courses. It makes the actors seem hopelessly small and fragile, and it’s menacing in a way that more futuristic, human-like designs wouldn’t be. I Am Mother isn’t an incredibly smart or memorable take on artificial intelligence, but the film still taps into some potent cultural anxieties.

What should it be rated?

At the upper end of PG-13. The film sets its apocalypse off-screen, and it focuses more on high tension than violent confrontation. But the human characters end up sporting a few bloody, painful-looking injuries.

How can I actually watch it?

I Am Mother will be released in Australia and New Zealand by StudioCanal at an unknown date; it appears to still be seeking an American distributor.

from The Verge http://bit.ly/1jLudMg
via IFTTT

Dolby made a secret app for recording studio quality audio on your phone

http://bit.ly/2T3G5QW

Dolby has been quietly testing a new mobile app for recording and cleaning up audio under the codename “234,” as first spotted by TechCrunch. The app, which was available through a website sign-up form, lets you record audio (a la Voice Notes), cancels background noise, and then apply presets, with names like “Amped,” “Thump,” and “Bright,” to theoretically make your recordings sound more professional.

The sign-up site, which has since been deactivated, advertised the app by saying: “How can music recorded on a phone sound so good? Dolby 234 automatically cleans up the sound, gives it tone and space, and finds the ideal loudness. It’s like having your own producer in your phone.” Those are big claims. Luckily, I snagged a copy of the app before the Dolby 234 site was deactivated and have been playing around with it to test it out.

Dolby says it’s “Like having your own producer in your phone”

The app itself is incredibly easy to use. Simply tap the record button and the app will measure room tone for a few seconds before starting the recording session. Once you stop recording, you can quickly polish the audio through adding a preset and tinkering with a few tools. Tool options include eliminating the room tone (unwanted background noise), adjusting the amount of bass and treble, adding “boost” (loudness), and trimming.

The app only comes with one preset but an “essentials” pack unlocks six more. These extra presets are available through a seven-day free trial and each is described with a few keywords. For example, “Lyric” is “full, smooth, and balanced,” while “Thump” is “deep, full, and powerful.” Unfortunately, you can’t see what’s under the hood for each preset. It only lets you adjust “intensity,” which is how much of the audio signal is sent through and affected. Once you’ve finished adjusting the recorded audio, you can rename it, and choose to share it to Dolby or to SoundCloud.

Image: Dolby

I’m usually very skeptical of one-button solutions for fixing audio, so I was eager to see if Dolby’s app could follow through on its claims. I recorded my voice with a window slightly open to create a more tonal background. The presets gave me varying success. “Standard” made me sound muddy, while “bright” put too much emphasis on plosives (popping sounds made from saying words with hard consonants, like “popping!”). There was one I liked — “deep” made my voice sound more full and widened.

The app is obviously a work in progress

The app looks glossy, but it’s obviously a work in progress. None of the presets worked enough magic to make my voice sound like it was recorded with a professional microphone. The mic level metering didn’t work either, failing to trigger when I recorded at high volumes. This is definitely not, as Dolby says, “like having your own producer in your phone.” At least not for now.

I’m also a confused on the audience for this. Why would Dolby make a consumer app for recording audio with an iPhone mic that uploads to SoundCloud, where the maximum sound resolution is 256kbps AAC? It also doesn’t have tremendous functionality. You can’t multitrack with it or adjust parameters within presets. The app feels very at odds with Dolby’s reputation as an expert in high-end audio quality. Slapping a compressor and de-esser on a recording can only do so much if you recorded with a mediocre mic in a noisy room. I certainly wouldn’t record a guitar this way and expect anything great, as the app shows in the introduction slides above.

But, who knows — maybe this is just the beginning and a more robust version of Dolby “234” is coming down the line. Regardless, the idea of an easy to use app that can polish up audio recorded on the go is appealing. Plus, it’s cool to see companies try to package professional tools for everyday creatives.

from The Verge http://bit.ly/1jLudMg
via IFTTT

jebmo’s posthaven